{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "copyright-notice",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Copyright 2017 Google LLC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "copyright-notice2",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PTaAdgy3LS8W",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " # 稀疏数据和嵌入简介\n",
    "\n",
    "**学习目标：**\n",
    "* 将影评字符串数据转换为稀疏特征矢量\n",
    "* 使用稀疏特征矢量实现情感分析线性模型\n",
    "* 通过将数据投射到二维空间的嵌入来实现情感分析 DNN 模型\n",
    "* 将嵌入可视化，以便查看模型学到的词语之间的关系\n",
    "\n",
    "在此练习中，我们将探讨稀疏数据，并使用影评文本数据（来自 [ACL 2011 IMDB 数据集](http://ai.stanford.edu/~amaas/data/sentiment/)）进行嵌入。这些数据已被处理成 `tf.Example` 格式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2AKGtmwNosU8",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 设置\n",
    "\n",
    "我们导入依赖项并下载训练数据和测试数据。[`tf.keras`](https://www.tensorflow.org/api_docs/python/tf/keras) 中包含一个文件下载和缓存工具，我们可以用它来检索数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "jGWqDqFFL_NZ",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\python\\framework\\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\python\\framework\\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\python\\framework\\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\python\\framework\\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\python\\framework\\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\python\\framework\\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorboard\\compat\\tensorflow_stub\\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorboard\\compat\\tensorflow_stub\\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorboard\\compat\\tensorflow_stub\\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorboard\\compat\\tensorflow_stub\\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorboard\\compat\\tensorflow_stub\\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "C:\\ProgramData\\Anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorboard\\compat\\tensorflow_stub\\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import collections\n",
    "import io\n",
    "import math\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from IPython import display\n",
    "from sklearn import metrics\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "train_url = 'https://download.mlcc.google.cn/mledu-datasets/sparse-data-embedding/train.tfrecord'\n",
    "train_path = tf.keras.utils.get_file(train_url.split('/')[-1], train_url)\n",
    "test_url = 'https://download.mlcc.google.cn/mledu-datasets/sparse-data-embedding/test.tfrecord'\n",
    "test_path = tf.keras.utils.get_file(test_url.split('/')[-1], test_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6W7aZ9qspZVj",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 构建情感分析模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jieA0k_NLS8a",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "总结:\n",
    "train.tfrecord 中的数据为一条一条的评论和他的结果: terms:list(每个单词隔开), label中为0.差评,1.好评\n",
    "terms_feature_column 为一个单词表,尽可能的包含训练数据里的所有单词(train.tfrecord),通过这个单词表把每一条评论映射为特征矢量(长度为单词表的长度)进行训练"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "我们根据这些数据训练一个情感分析模型，以预测**某条评价**总体上是*好评*（标签为 1）还是*差评*（标签为 0）。\n",
    "\n",
    "为此，我们会使用*词汇表*(terms_feature_column)(即我们预计将在数据(train.tfrecord)中看到的每个术语的列表(`字典`)），将字符串值 `terms` 转换为特征矢量(https://developers.google.cn/machine-learning/crash-course/embeddings/categorical-input-data) 。在本练习中，我们创建了侧重于一组有限术语的小型词汇表(terms_feature_column).其中的大多数术语明确表示是*好评*或*差评*，但有些只是因为有趣而被添加进来。\n",
    "\n",
    "词汇表中的每个术语都与特征矢量中的一个坐标相对应。为了将样本的字符串值 `terms` 转换为这种矢量格式，我们按以下方式处理字符串值：如果该术语没有出现在样本字符串中，则坐标值将为 0；如果出现在样本字符串中，则值为 1.(多热编码), 未出现在该词汇表中的样本中的术语将被弃用(全部为0的训练数据被弃用).\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2HSfklfnLS8b",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " **注意**：*我们当然可以使用更大的词汇表，而且有创建此类词汇表的专用工具。此外，我们可以添加少量的 OOV（未收录词汇）分桶，您可以在其中对词汇表中未包含的术语进行哈希处理，而不仅仅是弃用这些术语。我们还可以使用__特征哈希__法对每个术语进行哈希处理，而不是创建显式词汇表。这在实践中很有效，但却不具备可解读性（这对本练习非常实用）。如需了解处理此类词汇表的工具，请参阅 tf.feature_column 模块。*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Uvoa2HyDtgqe",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 构建输入管道"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O20vMEOurDol",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 首先，我们来配置输入管道，以将数据导入 TensorFlow 模型中。我们可以使用以下函数来解析训练数据和测试数据（格式为 [TFRecord](https://www.tensorflow.org/programmers_guide/datasets)），然后返回一个由特征和相应标签组成的字典。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "SxxNIEniPq2z",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def _parse_function(record):\n",
    "  \"\"\"Extracts features and labels.\n",
    "  \n",
    "  Args:\n",
    "    record: File path to a TFRecord file    \n",
    "  Returns:\n",
    "    A `tuple` `(labels, features)`:\n",
    "      features: A dict of tensors representing the features\n",
    "      labels: A tensor with the corresponding labels.\n",
    "  \"\"\"\n",
    "  features = {\n",
    "    \"terms\": tf.VarLenFeature(dtype=tf.string), # terms are strings of varying lengths\n",
    "    \"labels\": tf.FixedLenFeature(shape=[1], dtype=tf.float32) # labels are 0 or 1\n",
    "  }\n",
    "  \n",
    "  parsed_features = tf.parse_single_example(record, features)\n",
    "  \n",
    "  terms = parsed_features['terms'].values\n",
    "  labels = parsed_features['labels']\n",
    "  # print(\"test\", {'terms':terms}, labels)\n",
    "  return  {'terms':terms}, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SXhTeeYMrp-l",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 为了确认函数是否能正常运行，我们为训练数据构建一个 `TFRecordDataset`，并使用上述函数将数据映射到特征和标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "oF4YWXR0Omt0",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"IteratorGetNext:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_1:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_1:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_2:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_2:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_3:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_3:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_4:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_4:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_5:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_5:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_6:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_6:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_7:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_7:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_8:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_8:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_9:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_9:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_10:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_10:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_11:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_11:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_12:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_12:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_13:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_13:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_14:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_14:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_15:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_15:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_16:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_16:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_17:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_17:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_18:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_18:1\", shape=(1,), dtype=float32)\n",
      "Tensor(\"IteratorGetNext_19:0\", shape=(?,), dtype=string) Tensor(\"IteratorGetNext_19:1\", shape=(1,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Create the Dataset object.\n",
    "ds = tf.data.TFRecordDataset(train_path)\n",
    "# type(ds)\n",
    "# for train_data in ds[0:10]:\n",
    "#     print(train_data)\n",
    "# Map features and labels with the parse function.\n",
    "ds = ds.map(_parse_function)\n",
    "i = 0\n",
    "while i < 20:\n",
    "    features, labels = ds.make_one_shot_iterator().get_next()\n",
    "    print(features['terms'], labels)\n",
    "    i = i+1\n",
    "# ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bUoMvK-9tVXP",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 运行以下单元，以从训练数据集中获取第一个样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "Z6QE2DWRUc4E",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "({'terms': array([b'but', b'it', b'does', b'have', b'some', b'good', b'action',\n",
      "       b'and', b'a', b'plot', b'that', b'is', b'somewhat', b'interesting',\n",
      "       b'.', b'nevsky', b'acts', b'like', b'a', b'body', b'builder',\n",
      "       b'and', b'he', b'isn', b\"'\", b't', b'all', b'that', b'attractive',\n",
      "       b',', b'in', b'fact', b',', b'imo', b',', b'he', b'is', b'ugly',\n",
      "       b'.', b'(', b'his', b'acting', b'skills', b'lack', b'everything',\n",
      "       b'!', b')', b'sascha', b'is', b'played', b'very', b'well', b'by',\n",
      "       b'joanna', b'pacula', b',', b'but', b'she', b'needed', b'more',\n",
      "       b'lines', b'than', b'she', b'was', b'given', b',', b'her',\n",
      "       b'character', b'needed', b'to', b'be', b'developed', b'.',\n",
      "       b'there', b'are', b'way', b'too', b'many', b'men', b'in', b'this',\n",
      "       b'story', b',', b'there', b'is', b'zero', b'romance', b',', b'too',\n",
      "       b'much', b'action', b',', b'and', b'way', b'too', b'dumb', b'of',\n",
      "       b'an', b'ending', b'.', b'it', b'is', b'very', b'violent', b'.',\n",
      "       b'i', b'did', b'however', b'love', b'the', b'scenery', b',',\n",
      "       b'this', b'movie', b'takes', b'you', b'all', b'over', b'the',\n",
      "       b'world', b',', b'and', b'that', b'is', b'a', b'bonus', b'.', b'i',\n",
      "       b'also', b'liked', b'how', b'it', b'had', b'some', b'stuff',\n",
      "       b'about', b'the', b'mafia', b'in', b'it', b',', b'not', b'too',\n",
      "       b'much', b'or', b'too', b'little', b',', b'but', b'enough',\n",
      "       b'that', b'it', b'got', b'my', b'attention', b'.', b'the',\n",
      "       b'actors', b'needed', b'to', b'be', b'more', b'handsome', b'.',\n",
      "       b'.', b'.', b'the', b'biggest', b'problem', b'i', b'had', b'was',\n",
      "       b'that', b'nevsky', b'was', b'just', b'too', b'normal', b',',\n",
      "       b'not', b'sexy', b'enough', b'.', b'i', b'think', b'for', b'most',\n",
      "       b'guys', b',', b'sascha', b'will', b'be', b'hot', b'enough', b',',\n",
      "       b'but', b'for', b'us', b'ladies', b'that', b'are', b'fans', b'of',\n",
      "       b'action', b',', b'nevsky', b'just', b'doesn', b\"'\", b't', b'cut',\n",
      "       b'it', b'.', b'overall', b',', b'this', b'movie', b'was', b'fine',\n",
      "       b',', b'i', b'didn', b\"'\", b't', b'love', b'it', b'nor', b'did',\n",
      "       b'i', b'hate', b'it', b',', b'just', b'found', b'it', b'to', b'be',\n",
      "       b'another', b'normal', b'action', b'flick', b'.'], dtype=object)}, array([0.], dtype=float32))\n",
      "({'terms': array([b'but', b'it', b'does', b'have', b'some', b'good', b'action',\n",
      "       b'and', b'a', b'plot', b'that', b'is', b'somewhat', b'interesting',\n",
      "       b'.', b'nevsky', b'acts', b'like', b'a', b'body', b'builder',\n",
      "       b'and', b'he', b'isn', b\"'\", b't', b'all', b'that', b'attractive',\n",
      "       b',', b'in', b'fact', b',', b'imo', b',', b'he', b'is', b'ugly',\n",
      "       b'.', b'(', b'his', b'acting', b'skills', b'lack', b'everything',\n",
      "       b'!', b')', b'sascha', b'is', b'played', b'very', b'well', b'by',\n",
      "       b'joanna', b'pacula', b',', b'but', b'she', b'needed', b'more',\n",
      "       b'lines', b'than', b'she', b'was', b'given', b',', b'her',\n",
      "       b'character', b'needed', b'to', b'be', b'developed', b'.',\n",
      "       b'there', b'are', b'way', b'too', b'many', b'men', b'in', b'this',\n",
      "       b'story', b',', b'there', b'is', b'zero', b'romance', b',', b'too',\n",
      "       b'much', b'action', b',', b'and', b'way', b'too', b'dumb', b'of',\n",
      "       b'an', b'ending', b'.', b'it', b'is', b'very', b'violent', b'.',\n",
      "       b'i', b'did', b'however', b'love', b'the', b'scenery', b',',\n",
      "       b'this', b'movie', b'takes', b'you', b'all', b'over', b'the',\n",
      "       b'world', b',', b'and', b'that', b'is', b'a', b'bonus', b'.', b'i',\n",
      "       b'also', b'liked', b'how', b'it', b'had', b'some', b'stuff',\n",
      "       b'about', b'the', b'mafia', b'in', b'it', b',', b'not', b'too',\n",
      "       b'much', b'or', b'too', b'little', b',', b'but', b'enough',\n",
      "       b'that', b'it', b'got', b'my', b'attention', b'.', b'the',\n",
      "       b'actors', b'needed', b'to', b'be', b'more', b'handsome', b'.',\n",
      "       b'.', b'.', b'the', b'biggest', b'problem', b'i', b'had', b'was',\n",
      "       b'that', b'nevsky', b'was', b'just', b'too', b'normal', b',',\n",
      "       b'not', b'sexy', b'enough', b'.', b'i', b'think', b'for', b'most',\n",
      "       b'guys', b',', b'sascha', b'will', b'be', b'hot', b'enough', b',',\n",
      "       b'but', b'for', b'us', b'ladies', b'that', b'are', b'fans', b'of',\n",
      "       b'action', b',', b'nevsky', b'just', b'doesn', b\"'\", b't', b'cut',\n",
      "       b'it', b'.', b'overall', b',', b'this', b'movie', b'was', b'fine',\n",
      "       b',', b'i', b'didn', b\"'\", b't', b'love', b'it', b'nor', b'did',\n",
      "       b'i', b'hate', b'it', b',', b'just', b'found', b'it', b'to', b'be',\n",
      "       b'another', b'normal', b'action', b'flick', b'.'], dtype=object)}, array([0.], dtype=float32))\n",
      "({'terms': array([b'but', b'it', b'does', b'have', b'some', b'good', b'action',\n",
      "       b'and', b'a', b'plot', b'that', b'is', b'somewhat', b'interesting',\n",
      "       b'.', b'nevsky', b'acts', b'like', b'a', b'body', b'builder',\n",
      "       b'and', b'he', b'isn', b\"'\", b't', b'all', b'that', b'attractive',\n",
      "       b',', b'in', b'fact', b',', b'imo', b',', b'he', b'is', b'ugly',\n",
      "       b'.', b'(', b'his', b'acting', b'skills', b'lack', b'everything',\n",
      "       b'!', b')', b'sascha', b'is', b'played', b'very', b'well', b'by',\n",
      "       b'joanna', b'pacula', b',', b'but', b'she', b'needed', b'more',\n",
      "       b'lines', b'than', b'she', b'was', b'given', b',', b'her',\n",
      "       b'character', b'needed', b'to', b'be', b'developed', b'.',\n",
      "       b'there', b'are', b'way', b'too', b'many', b'men', b'in', b'this',\n",
      "       b'story', b',', b'there', b'is', b'zero', b'romance', b',', b'too',\n",
      "       b'much', b'action', b',', b'and', b'way', b'too', b'dumb', b'of',\n",
      "       b'an', b'ending', b'.', b'it', b'is', b'very', b'violent', b'.',\n",
      "       b'i', b'did', b'however', b'love', b'the', b'scenery', b',',\n",
      "       b'this', b'movie', b'takes', b'you', b'all', b'over', b'the',\n",
      "       b'world', b',', b'and', b'that', b'is', b'a', b'bonus', b'.', b'i',\n",
      "       b'also', b'liked', b'how', b'it', b'had', b'some', b'stuff',\n",
      "       b'about', b'the', b'mafia', b'in', b'it', b',', b'not', b'too',\n",
      "       b'much', b'or', b'too', b'little', b',', b'but', b'enough',\n",
      "       b'that', b'it', b'got', b'my', b'attention', b'.', b'the',\n",
      "       b'actors', b'needed', b'to', b'be', b'more', b'handsome', b'.',\n",
      "       b'.', b'.', b'the', b'biggest', b'problem', b'i', b'had', b'was',\n",
      "       b'that', b'nevsky', b'was', b'just', b'too', b'normal', b',',\n",
      "       b'not', b'sexy', b'enough', b'.', b'i', b'think', b'for', b'most',\n",
      "       b'guys', b',', b'sascha', b'will', b'be', b'hot', b'enough', b',',\n",
      "       b'but', b'for', b'us', b'ladies', b'that', b'are', b'fans', b'of',\n",
      "       b'action', b',', b'nevsky', b'just', b'doesn', b\"'\", b't', b'cut',\n",
      "       b'it', b'.', b'overall', b',', b'this', b'movie', b'was', b'fine',\n",
      "       b',', b'i', b'didn', b\"'\", b't', b'love', b'it', b'nor', b'did',\n",
      "       b'i', b'hate', b'it', b',', b'just', b'found', b'it', b'to', b'be',\n",
      "       b'another', b'normal', b'action', b'flick', b'.'], dtype=object)}, array([0.], dtype=float32))\n",
      "({'terms': array([b'but', b'it', b'does', b'have', b'some', b'good', b'action',\n",
      "       b'and', b'a', b'plot', b'that', b'is', b'somewhat', b'interesting',\n",
      "       b'.', b'nevsky', b'acts', b'like', b'a', b'body', b'builder',\n",
      "       b'and', b'he', b'isn', b\"'\", b't', b'all', b'that', b'attractive',\n",
      "       b',', b'in', b'fact', b',', b'imo', b',', b'he', b'is', b'ugly',\n",
      "       b'.', b'(', b'his', b'acting', b'skills', b'lack', b'everything',\n",
      "       b'!', b')', b'sascha', b'is', b'played', b'very', b'well', b'by',\n",
      "       b'joanna', b'pacula', b',', b'but', b'she', b'needed', b'more',\n",
      "       b'lines', b'than', b'she', b'was', b'given', b',', b'her',\n",
      "       b'character', b'needed', b'to', b'be', b'developed', b'.',\n",
      "       b'there', b'are', b'way', b'too', b'many', b'men', b'in', b'this',\n",
      "       b'story', b',', b'there', b'is', b'zero', b'romance', b',', b'too',\n",
      "       b'much', b'action', b',', b'and', b'way', b'too', b'dumb', b'of',\n",
      "       b'an', b'ending', b'.', b'it', b'is', b'very', b'violent', b'.',\n",
      "       b'i', b'did', b'however', b'love', b'the', b'scenery', b',',\n",
      "       b'this', b'movie', b'takes', b'you', b'all', b'over', b'the',\n",
      "       b'world', b',', b'and', b'that', b'is', b'a', b'bonus', b'.', b'i',\n",
      "       b'also', b'liked', b'how', b'it', b'had', b'some', b'stuff',\n",
      "       b'about', b'the', b'mafia', b'in', b'it', b',', b'not', b'too',\n",
      "       b'much', b'or', b'too', b'little', b',', b'but', b'enough',\n",
      "       b'that', b'it', b'got', b'my', b'attention', b'.', b'the',\n",
      "       b'actors', b'needed', b'to', b'be', b'more', b'handsome', b'.',\n",
      "       b'.', b'.', b'the', b'biggest', b'problem', b'i', b'had', b'was',\n",
      "       b'that', b'nevsky', b'was', b'just', b'too', b'normal', b',',\n",
      "       b'not', b'sexy', b'enough', b'.', b'i', b'think', b'for', b'most',\n",
      "       b'guys', b',', b'sascha', b'will', b'be', b'hot', b'enough', b',',\n",
      "       b'but', b'for', b'us', b'ladies', b'that', b'are', b'fans', b'of',\n",
      "       b'action', b',', b'nevsky', b'just', b'doesn', b\"'\", b't', b'cut',\n",
      "       b'it', b'.', b'overall', b',', b'this', b'movie', b'was', b'fine',\n",
      "       b',', b'i', b'didn', b\"'\", b't', b'love', b'it', b'nor', b'did',\n",
      "       b'i', b'hate', b'it', b',', b'just', b'found', b'it', b'to', b'be',\n",
      "       b'another', b'normal', b'action', b'flick', b'.'], dtype=object)}, array([0.], dtype=float32))\n",
      "({'terms': array([b'but', b'it', b'does', b'have', b'some', b'good', b'action',\n",
      "       b'and', b'a', b'plot', b'that', b'is', b'somewhat', b'interesting',\n",
      "       b'.', b'nevsky', b'acts', b'like', b'a', b'body', b'builder',\n",
      "       b'and', b'he', b'isn', b\"'\", b't', b'all', b'that', b'attractive',\n",
      "       b',', b'in', b'fact', b',', b'imo', b',', b'he', b'is', b'ugly',\n",
      "       b'.', b'(', b'his', b'acting', b'skills', b'lack', b'everything',\n",
      "       b'!', b')', b'sascha', b'is', b'played', b'very', b'well', b'by',\n",
      "       b'joanna', b'pacula', b',', b'but', b'she', b'needed', b'more',\n",
      "       b'lines', b'than', b'she', b'was', b'given', b',', b'her',\n",
      "       b'character', b'needed', b'to', b'be', b'developed', b'.',\n",
      "       b'there', b'are', b'way', b'too', b'many', b'men', b'in', b'this',\n",
      "       b'story', b',', b'there', b'is', b'zero', b'romance', b',', b'too',\n",
      "       b'much', b'action', b',', b'and', b'way', b'too', b'dumb', b'of',\n",
      "       b'an', b'ending', b'.', b'it', b'is', b'very', b'violent', b'.',\n",
      "       b'i', b'did', b'however', b'love', b'the', b'scenery', b',',\n",
      "       b'this', b'movie', b'takes', b'you', b'all', b'over', b'the',\n",
      "       b'world', b',', b'and', b'that', b'is', b'a', b'bonus', b'.', b'i',\n",
      "       b'also', b'liked', b'how', b'it', b'had', b'some', b'stuff',\n",
      "       b'about', b'the', b'mafia', b'in', b'it', b',', b'not', b'too',\n",
      "       b'much', b'or', b'too', b'little', b',', b'but', b'enough',\n",
      "       b'that', b'it', b'got', b'my', b'attention', b'.', b'the',\n",
      "       b'actors', b'needed', b'to', b'be', b'more', b'handsome', b'.',\n",
      "       b'.', b'.', b'the', b'biggest', b'problem', b'i', b'had', b'was',\n",
      "       b'that', b'nevsky', b'was', b'just', b'too', b'normal', b',',\n",
      "       b'not', b'sexy', b'enough', b'.', b'i', b'think', b'for', b'most',\n",
      "       b'guys', b',', b'sascha', b'will', b'be', b'hot', b'enough', b',',\n",
      "       b'but', b'for', b'us', b'ladies', b'that', b'are', b'fans', b'of',\n",
      "       b'action', b',', b'nevsky', b'just', b'doesn', b\"'\", b't', b'cut',\n",
      "       b'it', b'.', b'overall', b',', b'this', b'movie', b'was', b'fine',\n",
      "       b',', b'i', b'didn', b\"'\", b't', b'love', b'it', b'nor', b'did',\n",
      "       b'i', b'hate', b'it', b',', b'just', b'found', b'it', b'to', b'be',\n",
      "       b'another', b'normal', b'action', b'flick', b'.'], dtype=object)}, array([0.], dtype=float32))\n",
      "({'terms': array([b'but', b'it', b'does', b'have', b'some', b'good', b'action',\n",
      "       b'and', b'a', b'plot', b'that', b'is', b'somewhat', b'interesting',\n",
      "       b'.', b'nevsky', b'acts', b'like', b'a', b'body', b'builder',\n",
      "       b'and', b'he', b'isn', b\"'\", b't', b'all', b'that', b'attractive',\n",
      "       b',', b'in', b'fact', b',', b'imo', b',', b'he', b'is', b'ugly',\n",
      "       b'.', b'(', b'his', b'acting', b'skills', b'lack', b'everything',\n",
      "       b'!', b')', b'sascha', b'is', b'played', b'very', b'well', b'by',\n",
      "       b'joanna', b'pacula', b',', b'but', b'she', b'needed', b'more',\n",
      "       b'lines', b'than', b'she', b'was', b'given', b',', b'her',\n",
      "       b'character', b'needed', b'to', b'be', b'developed', b'.',\n",
      "       b'there', b'are', b'way', b'too', b'many', b'men', b'in', b'this',\n",
      "       b'story', b',', b'there', b'is', b'zero', b'romance', b',', b'too',\n",
      "       b'much', b'action', b',', b'and', b'way', b'too', b'dumb', b'of',\n",
      "       b'an', b'ending', b'.', b'it', b'is', b'very', b'violent', b'.',\n",
      "       b'i', b'did', b'however', b'love', b'the', b'scenery', b',',\n",
      "       b'this', b'movie', b'takes', b'you', b'all', b'over', b'the',\n",
      "       b'world', b',', b'and', b'that', b'is', b'a', b'bonus', b'.', b'i',\n",
      "       b'also', b'liked', b'how', b'it', b'had', b'some', b'stuff',\n",
      "       b'about', b'the', b'mafia', b'in', b'it', b',', b'not', b'too',\n",
      "       b'much', b'or', b'too', b'little', b',', b'but', b'enough',\n",
      "       b'that', b'it', b'got', b'my', b'attention', b'.', b'the',\n",
      "       b'actors', b'needed', b'to', b'be', b'more', b'handsome', b'.',\n",
      "       b'.', b'.', b'the', b'biggest', b'problem', b'i', b'had', b'was',\n",
      "       b'that', b'nevsky', b'was', b'just', b'too', b'normal', b',',\n",
      "       b'not', b'sexy', b'enough', b'.', b'i', b'think', b'for', b'most',\n",
      "       b'guys', b',', b'sascha', b'will', b'be', b'hot', b'enough', b',',\n",
      "       b'but', b'for', b'us', b'ladies', b'that', b'are', b'fans', b'of',\n",
      "       b'action', b',', b'nevsky', b'just', b'doesn', b\"'\", b't', b'cut',\n",
      "       b'it', b'.', b'overall', b',', b'this', b'movie', b'was', b'fine',\n",
      "       b',', b'i', b'didn', b\"'\", b't', b'love', b'it', b'nor', b'did',\n",
      "       b'i', b'hate', b'it', b',', b'just', b'found', b'it', b'to', b'be',\n",
      "       b'another', b'normal', b'action', b'flick', b'.'], dtype=object)}, array([0.], dtype=float32))\n"
     ]
    }
   ],
   "source": [
    "n = ds.make_one_shot_iterator().get_next()\n",
    "sess = tf.Session()\n",
    "res = sess.run(n)\n",
    "print(res)\n",
    "i = 0\n",
    "while i < 5:\n",
    "    n = ds.make_one_shot_iterator().get_next()\n",
    "    res = sess.run(n)\n",
    "    print(res)\n",
    "    i = i+1\n",
    "# n = ds.make_one_shot_iterator().get_next()\n",
    "# sess.run(n)\n",
    "# print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jBU39UeFty9S",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 现在，我们构建一个正式的输入函数，可以将其传递给 TensorFlow Estimator 对象的 `train()` 方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "5_C5-ueNYIn_",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Create an input_fn that parses the tf.Examples from the given files,\n",
    "# and split them into features and targets.\n",
    "def _input_fn(input_filenames, num_epochs=None, shuffle=True):\n",
    "  \n",
    "  # Same code as above; create a dataset and map features and labels.\n",
    "  ds = tf.data.TFRecordDataset(input_filenames)\n",
    "  ds = ds.map(_parse_function)\n",
    "\n",
    "  if shuffle:\n",
    "    ds = ds.shuffle(10000)\n",
    "\n",
    "  # Our feature data is variable-length, so we pad and batch\n",
    "  # each field of the dataset structure to whatever size is necessary.     \n",
    "  ds = ds.padded_batch(25, ds.output_shapes)\n",
    "  \n",
    "  ds = ds.repeat(num_epochs)\n",
    "\n",
    "  \n",
    "  # Return the next batch of data.\n",
    "  features, labels = ds.make_one_shot_iterator().get_next()\n",
    "  return features, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y170tVlrLS8c",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 任务 1：使用具有稀疏输入和显式词汇表的线性模型\n",
    "\n",
    "对于我们的第一个模型，我们将使用 50 个信息性术语来构建 [`LinearClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/LinearClassifier) 模型；始终从简单入手！\n",
    "\n",
    "以下代码将为我们的术语构建特征列。[`categorical_column_with_vocabulary_list`](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_list) 函数可使用“字符串-特征矢量”映射来创建特征列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "B5gdxuWsvPcx",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 50 informative terms that compose our model vocabulary.\n",
    "# 组成我们的模型词汇表的50个信息性术语。(通过这个词汇表把训练数据的每一个评论转换为多热编码的特征矢量(长度为词汇表长度)进行训练)\n",
    "informative_terms = (\"bad\", \"great\", \"best\", \"worst\", \"fun\", \"beautiful\",\n",
    "                     \"excellent\", \"poor\", \"boring\", \"awful\", \"terrible\",\n",
    "                     \"definitely\", \"perfect\", \"liked\", \"worse\", \"waste\",\n",
    "                     \"entertaining\", \"loved\", \"unfortunately\", \"amazing\",\n",
    "                     \"enjoyed\", \"favorite\", \"horrible\", \"brilliant\", \"highly\",\n",
    "                     \"simple\", \"annoying\", \"today\", \"hilarious\", \"enjoyable\",\n",
    "                     \"dull\", \"fantastic\", \"poorly\", \"fails\", \"disappointing\",\n",
    "                     \"disappointment\", \"not\", \"him\", \"her\", \"good\", \"time\",\n",
    "                     \"?\", \".\", \"!\", \"movie\", \"film\", \"action\", \"comedy\",\n",
    "                     \"drama\", \"family\")\n",
    "\n",
    "terms_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key=\"terms\", vocabulary_list=informative_terms)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eTiDwyorwd3P",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 接下来，我们将构建 `LinearClassifier`，在训练集中训练该模型，并在评估集中对其进行评估。阅读上述代码后，运行该模型以了解其效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "HYKKpGLqLS8d",
    "scrolled": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Entity <bound method LinearModel.call of <tensorflow.python.feature_column.feature_column_v2.LinearModel object at 0x000001E1C6C53EF0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LinearModel.call of <tensorflow.python.feature_column.feature_column_v2.LinearModel object at 0x000001E1C6C53EF0>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _LinearModelLayer.call of <tensorflow.python.feature_column.feature_column_v2._LinearModelLayer object at 0x000001E1C6CC9EB8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _LinearModelLayer.call of <tensorflow.python.feature_column.feature_column_v2._LinearModelLayer object at 0x000001E1C6CC9EB8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method LinearModel.call of <tensorflow.python.feature_column.feature_column_v2.LinearModel object at 0x000001E1C734A828>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LinearModel.call of <tensorflow.python.feature_column.feature_column_v2.LinearModel object at 0x000001E1C734A828>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _LinearModelLayer.call of <tensorflow.python.feature_column.feature_column_v2._LinearModelLayer object at 0x000001E1C73BB278>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _LinearModelLayer.call of <tensorflow.python.feature_column.feature_column_v2._LinearModelLayer object at 0x000001E1C73BB278>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Training set metrics:\n",
      "accuracy 0.78812\n",
      "accuracy_baseline 0.5\n",
      "auc 0.87188977\n",
      "auc_precision_recall 0.8619138\n",
      "average_loss 0.45141742\n",
      "label/mean 0.5\n",
      "loss 11.285436\n",
      "precision 0.75268364\n",
      "prediction/mean 0.518546\n",
      "recall 0.85824\n",
      "global_step 1000\n",
      "---\n",
      "WARNING: Entity <bound method LinearModel.call of <tensorflow.python.feature_column.feature_column_v2.LinearModel object at 0x000001E1CF0757B8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LinearModel.call of <tensorflow.python.feature_column.feature_column_v2.LinearModel object at 0x000001E1CF0757B8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _LinearModelLayer.call of <tensorflow.python.feature_column.feature_column_v2._LinearModelLayer object at 0x000001E1CF101D30>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _LinearModelLayer.call of <tensorflow.python.feature_column.feature_column_v2._LinearModelLayer object at 0x000001E1CF101D30>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Test set metrics:\n",
      "accuracy 0.7848\n",
      "accuracy_baseline 0.5\n",
      "auc 0.8704559\n",
      "auc_precision_recall 0.8599815\n",
      "average_loss 0.4515727\n",
      "label/mean 0.5\n",
      "loss 11.289317\n",
      "precision 0.7501054\n",
      "prediction/mean 0.5167597\n",
      "recall 0.85416\n",
      "global_step 1000\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "# 灵活地修改模型中每个系数的学习率，从而单调降低有效的学习率\n",
    "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
    "# 避免梯度爆炸\n",
    "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
    "\n",
    "feature_columns = [ terms_feature_column ]\n",
    "\n",
    "\n",
    "classifier = tf.estimator.LinearClassifier(\n",
    "  feature_columns=feature_columns,\n",
    "  optimizer=my_optimizer,\n",
    ")\n",
    "\n",
    "classifier.train(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "print(\"Training set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([test_path]),\n",
    "  steps=1000)\n",
    "\n",
    "print(\"Test set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J0ubn9gULS8g",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 任务 2：使用深度神经网络 (DNN) 模型\n",
    "\n",
    "上述模型是一个线性模型，效果非常好。但是，我们可以使用 DNN 模型实现更好的效果吗？\n",
    "\n",
    "我们将 `LinearClassifier` 切换为 [`DNNClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier)。运行以下单元，看看您的模型效果如何。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "jcgOPfEALS8h",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CE8973C8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CE8973C8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CEDD92B0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CEDD92B0>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CEDD97B8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CEDD97B8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CEDD9710>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CEDD9710>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CEDD9B00>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CEDD9B00>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CE897438>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CE897438>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CF17ABA8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CF17ABA8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF16F208>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF16F208>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF16F1D0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF16F1D0>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF16F550>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF16F550>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Training set metrics:\n",
      "accuracy 0.8\n",
      "accuracy_baseline 0.52\n",
      "auc 0.9423076\n",
      "auc_precision_recall 0.9528612\n",
      "average_loss 0.34197903\n",
      "label/mean 0.48\n",
      "loss 8.549476\n",
      "precision 0.7692308\n",
      "prediction/mean 0.49640283\n",
      "recall 0.8333333\n",
      "global_step 1000\n",
      "---\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D08C2B00>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D08C2B00>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D0942F28>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D0942F28>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D096D470>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D096D470>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D096D3C8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D096D3C8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D096D7B8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D096D7B8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Test set metrics:\n",
      "accuracy 0.72\n",
      "accuracy_baseline 0.52\n",
      "auc 0.7916666\n",
      "auc_precision_recall 0.8219237\n",
      "average_loss 0.5303837\n",
      "label/mean 0.48\n",
      "loss 13.259592\n",
      "precision 0.72727275\n",
      "prediction/mean 0.45779428\n",
      "recall 0.6666667\n",
      "global_step 1000\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "##################### Here's what we changed ##################################\n",
    "classifier = tf.estimator.DNNClassifier(                                      #\n",
    "  feature_columns=[tf.feature_column.indicator_column(terms_feature_column)], #\n",
    "  hidden_units=[20,20],                                                       #\n",
    "  optimizer=my_optimizer,                                                     #\n",
    ")                                                                             #\n",
    "###############################################################################\n",
    "\n",
    "try:\n",
    "  classifier.train(\n",
    "    input_fn=lambda: _input_fn([train_path]),\n",
    "    steps=1000)\n",
    "\n",
    "  evaluation_metrics = classifier.evaluate(\n",
    "    input_fn=lambda: _input_fn([train_path]),\n",
    "    steps=1)\n",
    "  print(\"Training set metrics:\")\n",
    "  for m in evaluation_metrics:\n",
    "    print(m, evaluation_metrics[m])\n",
    "  print(\"---\")\n",
    "\n",
    "  evaluation_metrics = classifier.evaluate(\n",
    "    input_fn=lambda: _input_fn([test_path]),\n",
    "    steps=1)\n",
    "\n",
    "  print(\"Test set metrics:\")\n",
    "  for m in evaluation_metrics:\n",
    "    print(m, evaluation_metrics[m])\n",
    "  print(\"---\")\n",
    "except ValueError as err:\n",
    "  print(err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cZz68luxLS8j",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 任务 3：在 DNN 模型中使用嵌入\n",
    "\n",
    "在此任务中，我们将使用嵌入列来实现 DNN 模型。嵌入(输入层(评论的特征矢量)后面加一层嵌入)列会将稀疏数据作为输入，并返回一个低维度密集矢量作为输出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AliRzhvJLS8k",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " **注意**：*从计算方面而言，embedding_column 通常是用于在稀疏数据中训练模型最有效的选项。在此练习末尾的[可选部分](#scrollTo=XDMlGgRfKSVz)，我们将更深入地讨论使用 `embedding_column` 与 `indicator_column` 之间的实现差异，以及如何在这两者之间做出权衡。*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F-as3PtALS8l",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 在下面的代码中，执行以下操作：\n",
    "\n",
    "* 通过将数据投射到二维空间的 `embedding_column` 来为模型定义特征列（如需详细了解 `embedding_column` 的函数签名，请参阅相关 [TF 文档](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column)）。\n",
    "* 定义符合以下规范的 `DNNClassifier`：\n",
    "  * 具有两个隐藏层，每个包含 20 个单元\n",
    "  * 采用学习速率为 0.1 的 AdaGrad 优化方法\n",
    "  * `gradient_clip_norm 值为 5.0`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UlPZ-Q9bLS8m",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " **注意**：*在实践中，我们可能会将数据投射到 2 维以上（比如 50 或 100）的空间中。但就目前而言，2 维是比较容易可视化的维数。*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mNCLhxsXyOIS",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ### 提示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "L67xYD7hLS8m",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Here's a example code snippet you might use to define the feature columns:\n",
    "# 下面是一个示例代码片段，您可以使用它来定义特性列:\n",
    "\n",
    "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
    "feature_columns = [ terms_embedding_column ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iv1UBsJxyV37",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ### 完成以下代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "5PG_yhNGLS8u",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "########################## YOUR CODE HERE ######################################\n",
    "# terms_embedding_column = # Define the embedding column\n",
    "# feature_columns = # Define the feature columns\n",
    "#\n",
    "# classifier = # Define the DNNClassifier\n",
    "# ################################################################################\n",
    "#\n",
    "# classifier.train(\n",
    "#   input_fn=lambda: _input_fn([train_path]),\n",
    "#   steps=1000)\n",
    "#\n",
    "# evaluation_metrics = classifier.evaluate(\n",
    "#   input_fn=lambda: _input_fn([train_path]),\n",
    "#   steps=1000)\n",
    "# print(\"Training set metrics:\")\n",
    "# for m in evaluation_metrics:\n",
    "#   print(m, evaluation_metrics[m])\n",
    "# print(\"---\")\n",
    "#\n",
    "# evaluation_metrics = classifier.evaluate(\n",
    "#   input_fn=lambda: _input_fn([test_path]),\n",
    "#   steps=1000)\n",
    "#\n",
    "# print(\"Test set metrics:\")\n",
    "# for m in evaluation_metrics:\n",
    "#   print(m, evaluation_metrics[m])\n",
    "# print(\"---\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D2BE3278>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D2BE3278>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D321F4E0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D321F4E0>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D321F5F8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D321F5F8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D321F898>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D321F898>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5DCDDD8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5DCDDD8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D715ACF8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D715ACF8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D54FADD8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D54FADD8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D3BF8320>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D3BF8320>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D3BF8278>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D3BF8278>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D3BF8668>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D3BF8668>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Training set metrics:\n",
      "accuracy 0.77944\n",
      "accuracy_baseline 0.5\n",
      "auc 0.86728656\n",
      "auc_precision_recall 0.85588616\n",
      "average_loss 0.46324122\n",
      "label/mean 0.5\n",
      "loss 11.58103\n",
      "precision 0.73674935\n",
      "prediction/mean 0.54958546\n",
      "recall 0.8696\n",
      "global_step 1000\n",
      "---\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D38A2198>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D38A2198>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D5DCDDA0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D5DCDDA0>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5553EB8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5553EB8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5DCDBE0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5DCDBE0>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5C42E10>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D5C42E10>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Test set metrics:\n",
      "accuracy 0.7796\n",
      "accuracy_baseline 0.5\n",
      "auc 0.86639893\n",
      "auc_precision_recall 0.8530096\n",
      "average_loss 0.4639417\n",
      "label/mean 0.5\n",
      "loss 11.598542\n",
      "precision 0.73820883\n",
      "prediction/mean 0.54795617\n",
      "recall 0.86648\n",
      "global_step 1000\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "########################## YOUR CODE HERE ######################################\n",
    "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)# Define the embedding column\n",
    "feature_columns = [ terms_embedding_column ]# Define the feature columns\n",
    "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
    "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
    "\n",
    "classifier = tf.estimator.DNNClassifier(\n",
    "  feature_columns=feature_columns,\n",
    "  optimizer=my_optimizer,\n",
    "  hidden_units=[20, 20]\n",
    ")\n",
    "# Define the DNNClassifier\n",
    "################################################################################\n",
    "\n",
    "classifier.train(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "print(\"Training set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([test_path]),\n",
    "  steps=1000)\n",
    "\n",
    "print(\"Test set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eQS5KQzBybTY",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ### 解决方案\n",
    "\n",
    "点击下方即可查看解决方案。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "R5xOdYeQydi5",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CF0AE7F0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CF0AE7F0>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CF171390>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CF171390>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF171828>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF171828>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF171940>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1CF171940>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D1405630>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D1405630>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CF5A29E8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1CF5A29E8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CF27AF60>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1CF27AF60>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D14334A8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D14334A8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D1433400>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D1433400>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D14337F0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D14337F0>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Training set metrics:\n",
      "accuracy 0.78652\n",
      "accuracy_baseline 0.5\n",
      "auc 0.8683444\n",
      "auc_precision_recall 0.8566883\n",
      "average_loss 0.45425108\n",
      "label/mean 0.5\n",
      "loss 11.356277\n",
      "precision 0.75838685\n",
      "prediction/mean 0.5131907\n",
      "recall 0.84096\n",
      "global_step 1000\n",
      "---\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D0F1EC50>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D0F1EC50>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D10ABB70>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D10ABB70>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D11CA0B8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D11CA0B8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D11CA080>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D11CA080>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D11CA400>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D11CA400>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Test set metrics:\n",
      "accuracy 0.78396\n",
      "accuracy_baseline 0.5\n",
      "auc 0.8683891\n",
      "auc_precision_recall 0.85608065\n",
      "average_loss 0.45379254\n",
      "label/mean 0.5\n",
      "loss 11.344813\n",
      "precision 0.7576395\n",
      "prediction/mean 0.5123108\n",
      "recall 0.83504\n",
      "global_step 1000\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "########################## SOLUTION CODE ########################################\n",
    "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
    "feature_columns = [ terms_embedding_column ]\n",
    "\n",
    "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
    "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
    "\n",
    "classifier = tf.estimator.DNNClassifier(\n",
    "  feature_columns=feature_columns,\n",
    "  hidden_units=[20,20],\n",
    "  optimizer=my_optimizer\n",
    ")\n",
    "#################################################################################\n",
    "\n",
    "classifier.train(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "print(\"Training set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([test_path]),\n",
    "  steps=1000)\n",
    "\n",
    "print(\"Test set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aiHnnVtzLS8w",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 任务 4：确信模型中确实存在嵌入\n",
    "\n",
    "上述模型使用了 `embedding_column`，而且似乎很有效，但这并没有让我们了解到内部发生的情形。我们如何检查该模型确实在内部使用了嵌入？\n",
    "\n",
    "首先，我们来看看该模型中的张量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "h1jNgLdQLS8w",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "['dnn/hiddenlayer_0/bias',\n 'dnn/hiddenlayer_0/bias/t_0/Adagrad',\n 'dnn/hiddenlayer_0/kernel',\n 'dnn/hiddenlayer_0/kernel/t_0/Adagrad',\n 'dnn/hiddenlayer_1/bias',\n 'dnn/hiddenlayer_1/bias/t_0/Adagrad',\n 'dnn/hiddenlayer_1/kernel',\n 'dnn/hiddenlayer_1/kernel/t_0/Adagrad',\n 'dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights',\n 'dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights/t_0/Adagrad',\n 'dnn/logits/bias',\n 'dnn/logits/bias/t_0/Adagrad',\n 'dnn/logits/kernel',\n 'dnn/logits/kernel/t_0/Adagrad',\n 'global_step']"
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.get_variable_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sl4-VctMLS8z",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 好的，我们可以看到这里有一个嵌入层：`'dnn/input_from_feature_columns/input_layer/terms_embedding/...'`。（顺便说一下，有趣的是，该层可以与模型的其他层一起训练，就像所有隐藏层一样。）\n",
    "\n",
    "嵌入层的形状是否正确？请运行以下代码来查明。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JNFxyQUiLS80",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " **注意**：*在我们的示例中，嵌入是一个矩阵，可让我们将一个 50 维矢量投射到 2 维空间。*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "1xMbpcEjLS80",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "(50, 2)"
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights').shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MnLCIogjLS82",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.007864   -0.9424145 ]\n",
      " [-0.8119308   0.19629993]\n",
      " [-0.9030074  -0.00275206]\n",
      " [ 2.4889219  -1.6050996 ]\n",
      " [-0.8903311  -0.23539948]\n",
      " [-0.4670383   0.38080445]\n",
      " [-0.9851294   1.553217  ]\n",
      " [ 1.1432896  -1.0115321 ]\n",
      " [ 1.4668628  -0.71298164]\n",
      " [ 1.3177278  -2.2157571 ]\n",
      " [ 1.5946686  -1.2055755 ]\n",
      " [-0.6903924   0.34454915]\n",
      " [-0.54982376  0.95706296]\n",
      " [ 0.3449111   1.293702  ]\n",
      " [ 0.8135228  -0.9660552 ]\n",
      " [ 1.099206   -2.180646  ]\n",
      " [ 0.03428723  0.8865632 ]\n",
      " [-0.9428589  -0.01318748]\n",
      " [ 0.88040996 -0.00786143]\n",
      " [-1.7524232   0.3533801 ]\n",
      " [-0.65469646  0.12081388]\n",
      " [-1.1008102   0.38304698]\n",
      " [ 1.4659883  -0.72766197]\n",
      " [-1.2192436  -0.34631926]\n",
      " [-1.775212   -0.52101576]\n",
      " [-0.43355593  1.3483278 ]\n",
      " [ 1.3090494  -0.22677067]\n",
      " [-1.5365158   0.20854574]\n",
      " [-0.76713824 -0.30737206]\n",
      " [-0.503919    0.31861073]\n",
      " [ 1.3277187  -0.5141243 ]\n",
      " [-1.2958765   0.2695841 ]\n",
      " [ 1.9157305  -0.50277627]\n",
      " [ 1.0252026  -1.2877712 ]\n",
      " [ 0.9928169  -0.60637516]\n",
      " [ 0.4989459  -1.6972847 ]\n",
      " [ 0.52643025  0.56618696]\n",
      " [-0.24279557  0.06197056]\n",
      " [ 0.01470514  0.08415478]\n",
      " [ 0.38074797  0.9508618 ]\n",
      " [ 0.03014935  0.09411182]\n",
      " [-0.02095743 -1.2006774 ]\n",
      " [-0.06390218 -0.2555608 ]\n",
      " [ 0.6184537   0.92612314]\n",
      " [-0.3418941  -0.83631337]\n",
      " [-0.09426594 -0.21188185]\n",
      " [-0.3265625  -0.34985998]\n",
      " [ 0.5782643   0.75073177]\n",
      " [-0.52384317 -0.5360646 ]\n",
      " [ 0.17465048  0.6713496 ]]\n"
     ]
    }
   ],
   "source": [
    "embedding = classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights')\n",
    "# 训练后嵌套层就是这50个词的二维关系映射\n",
    "print(embedding);"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    " 花些时间来手动检查各个层及其形状，以确保一切都按照您预期的方式互相连接。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rkKAaRWDLS83",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 任务 5：检查嵌入\n",
    "\n",
    "现在，我们来看看实际嵌入空间，并了解术语最终所在的位置。请执行以下操作：\n",
    "1. 运行以下代码来查看我们在**任务 3** 中训练的嵌入。一切最终是否如您所预期的那样？\n",
    "\n",
    "2. 重新运行**任务 3** 中的代码来重新训练该模型，然后再次运行下面的嵌入可视化。哪些保持不变？哪些发生了变化？\n",
    "\n",
    "3. 最后，仅使用 10 步来重新训练该模型（这将产生一个糟糕的模型）。再次运行下面的嵌入可视化。您现在看到了什么？为什么？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "s4NNu7KqLS84",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 1080x1080 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3IAAANOCAYAAABUbz43AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde1RVdf7/8dcWDO+gozOjjiPWaF7gcICjIMjNvOWg5S0tnSCXGpXWsrF0ssgcayoZNZuK0TGxxsjCe9nUaKKYOAJ6RHTSohjtq1M6CkGAgu7fH+b5ecNbyHHD87GWa5392Xt/9vtzai19nc9n722YpikAAAAAgHXUc3cBAAAAAIBrQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDGe7i7gclq2bGn6+vq6uwwAAAAAcIucnJyjpmm2urD9pg5yvr6+ys7OdncZAAAAAOAWhmH851LtLK0EAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwAAAAAWQ5ADAAAAAIupliBnGMabhmF8ZxhGXhX7ow3DKDIMw/njn8TquC4AAAAA1EWe1dRPiqS/SHrrMsdkmKYZW03XAwAAAIA6q1pm5EzT3CzpWHX0BQAAAAC4vJq8R66nYRi7DMP4yDCMblUdZBjGBMMwsg3DyD5y5EgNlgcAAAAA1lBTQW6HpPamaQZIelXSqqoONE1zgWmaDtM0Ha1ataqh8gAAAADAOmokyJmm+b1pmiU/fl4nqb5hGC1r4toAAAAAUNvUSJAzDOOXhmEYP37u8eN1/1cT1wYAAACA2qZanlppGEaqpGhJLQ3D+EbSs5LqS5JpmsmShkt6yDCMSkllkkaZpmlWx7UBAAAAoK6pliBnmua9V9j/F515PQEAAAAA4CeqyadWAgAAAACqAUEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAOAyY8YMJSUlubsMAMAVEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAuMyYMUNTpkxxdxkAfqIrPbgoPT1dsbGxkqSUlBRNnDixpkpDNSHIAQAAAIDFEOQAAIBLcnKy3nrrLXeXAeA6PP/887r99tvVp08f7du3T5IUHR2t7OxsSdLRo0fl6+vrxgpRnTzdXQAAALh5JCQkuLsEANchJydH7777rnbu3KnKykoFBQUpODjY3WXhBiLIAQAAABaXkZGhIUOGqFGjRpKkwYMHu7ki3GgsrQQAAABqAcMwLmrz9PTU6dOnJUnl5eU1XRJuIIIcAAAAYHGRkZFauXKlysrKVFxcrLVr10qSfH19lZOTI0lKS0tzZ4moZgQ5AAAAwOKCgoI0cuRI2e12DRs2TBEREZKkKVOm6I033lBYWJiOHj3q5ipRnQzTNN1dQ5UcDod59ik7AAAAAFDXGIaRY5qm48J2ZuQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAABgMQQ5AAAAALAYghwAAAAAWAxBDgAAAAAshiAHAAAAABZDkAMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAoM4qKCiQn5/fDe3/nXfecW1nZ2fr0UcflSSdOHFCffr0kd1u17Jly6rsIyUlRRMnTrxhNQKwJk93FwAAAFBbnQ1y9913nyTJ4XDI4XBIknbu3KmKigo5nU53lgjAopiRAwAAdVplZaXi4uJks9k0fPhwlZaWKicnR1FRUQoODlb//v11+PBhSdLChQvVvXt3BQQEaNiwYSotLZUkxcfHKy0tzdVnkyZNJEnTpk1TRkaG7Ha75s6dq/T0dMXGxuq7777TmDFj5HQ6ZbfblZ+fL19fXx09elTSmZm76Ojomv0iAFgKQQ4AANRp+/bt04QJE5Sbm6tmzZrptdde06RJk5SWlqacnByNHTtW06dPlyQNHTpUWVlZ2rVrl7p06aJFixZdtu8XX3xRERERcjqdmjx5sqv95z//uf72t7+59t122203dIwAah+WVgIAgDqtXbt2Cg8PlySNGTNGL7zwgvLy8tS3b19J0qlTp9S6dWtJUl5enp5++mkVFhaqpKRE/fv3d1vdAOo2ghwAAKjTDMM4b7tp06bq1q2bMjMzLzo2Pj5eq1atUkBAgFJSUpSeni5J8vT01OnTpyVJpmnq5MmT11zHuX2Ul5df8/kA6haWVgIAgDrtwIEDrtCWmpqq0NBQHTlyxNVWUVGhPXv2SJKKi4vVunVrVVRUaOnSpa4+fH19lZOTI0lavXq1KioqJJ0JhcXFxVdVx7l9LF++vHoGB6DWIsgBAIA6rUuXLlqyZIlsNpuOHTvmuj9u6tSpCggIkN1u19atWyVJf/zjHxUSEqK+ffuqc+fOrj7Gjx+vTZs2qUePHvrXv/6lxo0bS5JsNps8PT0VEBCguXPnXraOZ599Vo899pgiIiLk4eFx4wYMoFYwTNN0dw1VcjgcZnZ2trvLAAAAAAC3MAwjxzRNx4XtzMgBAAAAgMUQ5AAAAADAYghyAAAAAGAx1RLkDMN40zCM7wzDyKtiv2EYxnzDML40DCPXMIyg6rguAAAAANRF1TUjlyJpwGX23ymp449/Jkh6o5quCwAAAAB1TrUEOdM0N0s6dplD7pL0lnnGNkk+hmG0ro5rAwAAAEBdU1P3yLWVdPCc7W9+bLuIYRgTDMPINgwj+8iRIzVSHAAAAABYSU0FOeMSbZd8gZ1pmgtM03SYpulo1arVDS4LAAAAAKynpoLcN5LanbP9K0mHaujaAAAAAFCr1FSQWyPp/h+fXhkqqcg0zcM1dG0AAAAAqFU8q6MTwzBSJUVLamkYxjeSnpVUX5JM00yWtE7SQElfSiqV9EB1XBcAAAAA6qJqCXKmad57hf2mpEeq41oAAAAAUNfV1NJKAAAAAEA1IcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAN8TAgQN16NAhd5cBAABQK3m6uwAAtdO6devcXQIAAECtxYwcAAAAAMtJSUmp06t/CHIAAAAALIcgBwAAAABuVlBQoC5dumj8+PHq1q2b+vXrp7KyMjmdToWGhspms2nIkCE6fvy40tLSlJ2drdGjR8tut6usrMzd5dc4ghwAAACAm8IXX3yhRx55RHv27JGPj4+WL1+u+++/Xy+99JJyc3Pl7++v5557TsOHD5fD4dDSpUvldDrVsGFDd5de4whyAAAAAG4KHTp0kN1ulyQFBwcrPz9fhYWFioqKkiTFxcVp8+bN7izxpkGQAwAAAHBT8PLycn328PBQYWGhG6u5uRHkAAAAANyUvL291bx5c2VkZEiS3n77bdfsXNOmTVVcXOzO8tyK98gBAAAAuGktWbJECQkJKi0t1a233qrFixdLkuLj45WQkKCGDRsqMzOzzt0nZ5im6e4aquRwOMzs7Gx3lwEAAAAAbmEYRo5pmo4L21laCQAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ6WNH/+fHXp0kWjR4++5P7s7Gw9+uijkqSUlBRNnDixJssDAAAAbihPdxcAXI/XX39dH330kTp06HDJ/Q6HQw7HRe9NBAAAAGoFZuRgOQkJCfrqq680ePBgvfTSSwoLC1NgYKDCwsK0b98+SVJ6erpiY2MvOvf999+Xn5+fAgICFBkZWdOlAwAAANWCGTlYTnJysv7xj39o48aNuuWWW/T73/9enp6eWr9+vZ566iktX768ynNnzpypjz/+WG3btlVhYWENVg0AAABUH4IcLK2oqEhxcXH64osvZBiGKioqLnt8eHi44uPjdc8992jo0KE1VCUAAABQvVhaCUt75plnFBMTo7y8PK1du1bl5eWXPT45OVmzZs3SwYMHZbfb9b///a+GKgVws0hPT9fWrVtd28nJyXrrrbfcWBEAANeOGTlYWlFRkdq2bSvpzNMpryQ/P18hISEKCQnR2rVrdfDgQf3sZz+7wVUCuJmkp6erSZMmCgsLk3TmvlsAAKyGGTlY2pNPPqk//OEPCg8P16lTp654/BNPPCF/f3/5+fkpMjJSAQEBNVAlarsZM2YoKSlJiYmJWr9+vVtrGThw4BXv/0xJSdGhQ4dc2+PGjdPevXtvdGk33N13363g4GB169ZNCxYskCT94x//UFBQkAICAnTHHXeooKBAycnJmjt3rux2uzIyMlz//STJ6XQqNDRUNptNQ4YM0fHjxyVJ0dHRmjp1qnr06KFOnTopIyPDbeMEAECSDNM03V1DlRwOh5mdne3uMgDgsmbMmKEmTZpoypQp7i7lqkRHRyspKanWvaLj2LFjatGihcrKytS9e3dt2LBBDodDmzdvVocOHVz7L/zvde62zWbTq6++qqioKCUmJur777/XvHnzFB0dreDgYP35z3/WunXrNGfOHLeHdgBA3WAYRo5pmhf9pc2MHABch+eff1633367+vTp43rtRXx8vNLS0iRJ06ZNU9euXWWz2VyBYe3atQoJCVFgYKD69Omjb7/9VtKZIPG73/1OvXv3VseOHbVw4UJJZ5YARkZGasiQIeratasSEhJ0+vRpSVJqaqprdnnq1Kmuunx9fXX06FEVFBSoS5cuGj9+vLp166Z+/fqprKxMaWlpys7O1ujRo2W321VWVqbo6Gid/dGsSZMmmj59ugICAhQaGuqqMT8/X6GhoerevbsSExPVpEmTGviWr838+fNddR88eFALFixQZGSk632TLVq0uOz5RUVFKiwsVFRUlCQpLi5Omzdvdu0/+4Ck4OBgFRQU3JhBAABwlQhyAHCNcnJy9O6772rnzp1asWKFsrKyztt/7NgxrVy5Unv27FFubq6efvppSVKvXr20bds27dy5U6NGjdLLL7/sOic3N1cffvihMjMzNXPmTNfSx+3bt+vPf/6zdu/erfz8fK1YsUKHDh3S1KlT9emnn8rpdCorK0urVq26qM4vvvhCjzzyiPbs2SMfHx8tX75cw4cPl8Ph0NKlS+V0OtWwYcPzzvnhhx8UGhqqXbt2KTIy0hUqH3vsMT322GPKyspSmzZtqvX7rA7p6elav369MjMztWvXLgUGBiogIECGYVTbNby8vCRJHh4eqqysrLZ+AQC4HgQ5ALhGGRkZGjJkiBo1aqRmzZpp8ODB5+1v1qyZGjRooHHjxmnFihVq1KiRJOmbb75R//795e/vr9mzZ2vPnj2uc+666y41bNhQLVu2VExMjLZv3y5J6tGjh2699VZ5eHjo3nvv1ZYtW5SVlaXo6Gi1atVKnp6eGj169HkzR2d16NBBdrtd0tXPIt1yyy2KjY296JzMzEyNGDFCknTfffdd2xdWA4qKitS8eXM1atRIn3/+ubZt26YTJ05o06ZN+vrrryWdCdiS1LRpUxUXF1/Uh7e3t5o3b+66/+3tt992zc4BAHCzIcgBwHW43EyPp6entm/frmHDhmnVqlUaMGCAJGnSpEmaOHGidu/erb/+9a/nvS7jwv7Obl+q/WrvbT47gyRd/SxS/fr1Xde00szTgAEDVFlZKZvNpmeeeUahoaFq1aqVFixYoKFDhyogIEAjR46UJA0aNEgrV650PezkXEuWLNETTzwhm80mp9OpxMREdwwHAIAr4vUDAHCNIiMjFR8fr2nTpqmyslJr167Vgw8+6NpfUlKi0tJSDRw4UKGhofrNb34j6fzXZSxZsuS8PlevXq0//OEP+uGHH5Senq4XX3xR+/fv1/bt2/X111+rffv2WrZsmSZMmKCQkBA99thjOnr0qJo3b67U1FRNmjTpquuvakbqckJDQ7V8+XKNHDlS77777jWdWxO8vLz00UcfXXLfnXfeed52p06dlJub69qOiIhwfbbb7dq2bdtFfaSnp7s+t2zZknvkAABuR5ADgGsUFBSkkSNHym63q3379ucFAUkqLi7WXXfdpfLycpmmqblz50o681CTESNGqG3btgoNDXUt+ZPOLKH87W9/qwMHDuiZZ55RmzZttH//fvXs2VPTpk3T7t27XQ8+qVevnv70pz8pJiZGpmlq4MCBuuuuu666/vj4eCUkJKhhw4bKzMy8qnPmzZunMWPG6M9//rN++9vfytvb+6qvBwAAqh+vHwAAN6vq9QXp6elKSkrSBx984KbK/r/S0lI1bNhQhmHo3XffVWpqqlavXu3usgAAqPWqev0AM3IAgCvKycnRxIkTZZqmfHx89Oabb7q7JAAA6jRm5AAAAADgJsULwQEAAACgliDIAQAAAIDFEOQAAAAAwGIIcgAAALVIQUGB/Pz8avxcADWLIAcAAAAAFkOQAwAAqGUqKysVFxcnm82m4cOHq7S0VDNnzlT37t3l5+enCRMm6OyTy3NychQQEKCePXvqtddec3PlAK4WQQ4AAKCW2bdvnyZMmKDc3Fw1a9ZMr7/+uiZOnKisrCzl5eWprKxMH3zwgSTpgQce0Pz585WZmenmqgFcC4IcAABALdOuXTuFh4dLksaMGaMtW7Zo48aNCgkJkb+/vz799FPt2bNHRUVFKiwsVFRUlCTpd7/7nTvLBnANPN1dAAAAAKqXYRgXbT/88MPKzs5Wu3btNGPGDJWXl8s0zYuOBWANzMgBAADUMgcOHHAtlUxNTVWvXr0kSS1btlRJSYnS0tIkST4+PvL29taWLVskSUuXLnVPwQCuGTNyAAAAtUyXLl20ZMkSPfjgg+rYsaMeeughHT9+XP7+/vL19VX37t1dxy5evFhjx45Vo0aN1L9/fzdWDeBaGGefWHQzcjgcZnZ2trvLAAAAAAC3MAwjxzRNx4XtLK0EAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAzlFQUCA/P7/rPj8xMVHr16+vxooAALgYrx8AAKCanDp1SjNnznR3GQCAOoAZOQAALlBZWam4uDjZbDYNHz5cpaWl2rBhgwIDA+Xv76+xY8fqxIkTkiRfX1/NnDlTvXr10vvvv6/4+HjXy5Z9fX317LPPKigoSP7+/vr8888lSUeOHFHfvn0VFBSkBx98UO3bt9fRo0fdNl4AgPUQ5AAAuMC+ffs0YcIE5ebmqlmzZpozZ47i4+O1bNky7d69W5WVlXrjjTdcxzdo0EBbtmzRqFGjLuqrZcuW2rFjhx566CElJSVJkp577jn17t1bO3bs0JAhQ3TgwIEaGxsAoHYgyAEAcIF27dopPDxckjRmzBht2LBBHTp0UKdOnSRJcXFx2rx5s+v4kSNHVtnX0KFDJUnBwcEqKCiQpPNC34ABA9S8efMbMQwAQC1GkAMA4AKGYVzT8Y0bN65yn5eXlyTJw8NDlZWVkiTTNK+/OAAARJADAOAiBw4cUGZmpiQpNTVVffr0UUFBgb788ktJ0ttvv62oqKjr7r9Xr1567733JEmffPKJjh8//tOLBgDUKQQ5AAAu0KVLFy1ZskQ2m03Hjh3T5MmTtXjxYo0YMUL+/v6qV6+eEhISrrv/Z599Vp988omCgoL00UcfqXXr1mratGk1jgAAUNsZN/PyDofDYWZnZ7u7DAAAqtWJEyfk4eEhT09PZWZm6qGHHpLT6XR3WQCAm5BhGDmmaToubOc9cgAA1LADBw7onnvu0enTp3XLLbdo4cKF7i4JAGAxBDkAAGpYx44dtXPnTneXAQCwMO6RAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYDEEOAAAAACyGIAcAAAAAFkOQAwAAAACLIcgBAAAAgMUQ5AAAAADAYghyAAAAAGAxBDkAAAAAsBiCHAAAAABYTLUEOcMwBhiGsc8wjC8Nw5h2if3RhmEUGYbh/PFPYnVcFwAAAADqIs+f2oFhGB6SXpPUV9I3krIMw1hjmubeCw7NME0z9qdeDwAAAADquuqYkesh6UvTNL8yTfOkpHcl3VUN/QIAAAAALqE6glxbSQfP2f7mx7YL9TQMY5dhGB8ZhtGtGq4LAAAAAHXST15aKcm4RJt5wfYOSe1N0ywxDGOgpFWSOl6yM8OYIGmCJP3617+uhvIAAAAAoHapjhm5byS1O2f7V5IOnXuAaZrfm6ZZ8uPndZLqG4bR8lKdmaa5wDRNh2majlatWlVDeQAAAABQu1RHkMuS1NEwjA6GYdwiaZSkNeceYBjGLw3DMH783OPH6/6vGq4NAAAAAHXOT15aaZpmpWEYEyV9LMlD0pumae4xDCPhx/3JkoZLesgwjEpJZZJGmaZ54fJLAAAAAMBVMG7mPOVwOMzs7Gx3lwEAAAAAbmEYRo5pmo4L26vlheAAAAAAgJpDkAMAAAAAiyHIAedYtWqV9u7d6+4yAAAAgMsiyKFOOnXq1CXbCXIAAACwAoIcLOfll1/W/PnzJUmTJ09W7969JUkbNmzQmDFjlJqaKn9/f/n5+Wnq1Kmu85o0aaLExESFhIQoMzNT06ZNU9euXWWz2TRlyhRt3bpVa9as0RNPPCG73a78/Hy3jA8AAAC4EoIcLCcyMlIZGRmSpOzsbJWUlKiiokJbtmxRx44dNXXqVH366adyOp3KysrSqlWrJEk//PCD/Pz89K9//Utdu3bVypUrtWfPHuXm5urpp59WWFiYBg8erNmzZ8vpdOq2225z5zABAACAKhHkYDnBwcHKyclRcXGxvLy81LNnT2VnZysjI0M+Pj6Kjo5Wq1at5OnpqdGjR2vz5s2SJA8PDw0bNkyS1KxZMzVo0EDjxo3TihUr1KhRI3cOCQAAALgmBDlYTv369eXr66vFixcrLCxMERER2rhxo/Lz8/XrX/+6yvMaNGggDw8PSZKnp6e2b9+uYcOGadWqVRowYEBNlQ8AAAD8ZAQ5WFJkZKSSkpIUGRmpiIgIJScny263KzQ0VJs2bdLRo0d16tQppaamKioq6qLzS0pKVFRUpIEDB2revHlyOp2SpKZNm6q4uLimhwMAAABcE4IcLCkiIkKHDx9Wz5499Ytf/EINGjRQRESEWrdurT/96U+KiYlRQECAgoKCdNddd110fnFxsWJjY2Wz2RQVFaW5c+dKkkaNGqXZs2crMDCQh50AAADgpmWYpunuGqrkcDjM7Oxsd5cBAAAAAG5hGEaOaZqOC9uZkQMAAAAAiyHIAQAAAIDFEOQAAAAAwGIIcgAAAHXA/Pnz1aVLF40ePfon9ZOYmKj169dLkqKjo8XzDAD38HR3AQAAALjxXn/9dX300Ufq0KHDT+pn5syZ1VQRgJ+CGTkAAIBaLiEhQV999ZUGDx6sl156SWFhYQoMDFRYWJj27dsnSUpJSdHdd9+tQYMGqUOHDvrLX/6iOXPmKDAwUKGhoTp27JgkKT4+Xmlpaef1v2jRIk2ePNm1vXDhQj3++OM1N0CgDiLIAUAtVVBQID8/v5/UR3p6urZu3VpNFQFwl+TkZLVp00YbN27UQw89pM2bN2vnzp2aOXOmnnrqKddxeXl5euedd7R9+3ZNnz5djRo10s6dO9WzZ0+99dZbVfY/atQorVmzRhUVFZKkxYsX64EHHrjh4wLqMpZWAgCqlJ6eriZNmigsLMzdpQCoJkVFRYqLi9MXX3whwzBc4UuSYmJi1LRpUzVt2lTe3t4aNGiQJMnf31+5ublV9ukCe+EAACAASURBVNm4cWP17t1bH3zwgbp06aKKigr5+/vf8LEAdRkzcgBQi1VWViouLk42m03Dhw9XaWmpcnJyFBUVpeDgYPXv31+HDx+WdOZBCF27dpXNZtOoUaNUUFCg5ORkzZ07V3a7XRkZGW4eDYDq8MwzzygmJkZ5eXlau3atysvLXfu8vLxcn+vVq+farlevniorKy/b77hx45SSksJsHFBDmJEDgFps3759WrRokcLDwzV27Fi99tprWrlypVavXq1WrVpp2bJlmj59ut588029+OKL+vrrr+Xl5aXCwkL5+PgoISFBTZo00ZQpU9w9FADVpKioSG3btpV05r646hISEqKDBw9qx44dl529A1A9mJEDgFqsXbt2Cg8PlySNGTNGH3/8sfLy8tS3b1/Z7XbNmjVL33zzjSTJZrNp9OjR+vvf/y5PT37nA2qrJ598Un/4wx8UHh6uU6dOVWvf99xzj8LDw9W8efNq7RfAxQzTNN1dQ5UcDofJu0kA4PoUFBQoKipK//nPfyRJn376qV599VX997//VWZm5kXHnzp1Sps3b9aaNWu0bt067dmzR7NmzWJGDsBVi42N1eTJk3XHHXe4uxSg1jAMI8c0TceF7czIAUAtduDAAVdoS01NVWhoqI4cOeJqq6io0J49e3T69GkdPHhQMTExevnll1VYWKiSkhI1bdpUxcXF7hwCAAsoLCxUp06d1LBhQ0IcUEMIcgBQi3Xp0kVLliyRzWbTsWPHNGnSJKWlpWnq1KkKCAiQ3W7X1q1bderUKY0ZM0b+/v4KDAzU5MmT5ePjo0GDBmnlypU87ATAZfn4+Gj//v16//333V0KUGewtBIAAAAAblIsrQQAAACAWoIgBwAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQCoYadOnXJ3CQAAwOIIcgBwDV5++WXNnz9fkjR58mT17t1bkrRhwwaNGTNGqamp8vf3l5+fn6ZOneo6r0mTJkpMTFRISIgyMzM1bdo0de3aVTabTVOmTJEkHTlyRMOGDVP37t3VvXt3ffbZZzU/QNQ5hYWFev311yVJhw4d0vDhw91cEQDgahDkAOAaREZGKiMjQ5KUnZ2tkpISVVRUaMuWLerYsaOmTp2qTz/9VE6nU1lZWVq1apUk6YcffpCfn5/+9a9/qWvXrlq5cqX27Nmj3NxcPf3005Kkxx57TJMnT9b777+v77//XuPGjTvv2omJiVq/fn3NDhi13rlBrk2bNkpLS3NzRQCAq+Hp7gIAwEqCg4OVk5Oj4uJieXl5KSgoSNnZ2crIyNCgQYMUHR2tVq1aSZJGjx6tzZs36+6775aHh4eGDRsmSWrWrJkaNGigcePG6be//a1iY2MlSevXr9fevXt18uRJHThwQC1atFBxcbGaNm0qSZo5c6Z7Bo1abdq0acrPz5fdblfHjh3173//W3l5eUpJSdGqVat06tQp5eXl6fe//71Onjypt99+W15eXlq3bp1atGih/Px8PfLIIzpy5IgaNWqkhQsXqnPnzu4eFgDUeszIAcA1qF+/vnx9fbV48WKFhYUpIiJCGzduVH5+vn79619XeV6DBg3k4eEhSfL09NT27ds1bNgwrVq1SgMGDJAknT59WpmZmVq3bp18fX01cOBAhYaGql+/fiorK1N8fLxrtsTX11dPPfWUevbsKYfDoR07dqh///667bbblJycfOO/CNQaL774om677TY5nU7Nnj37vH15eXl65513tH37dk2fPl2NGjXSzp071bNnT7311luSpAkTJujVV19VTk6OkpKS9PDDD7tjGABQ5xDkAOAaRUZGKikpSZGRkYqIiFBycrLsdrtCQ0O1adMmHT16VKdOnVJqaqqioqIuOr+kpERFRUUaOHCg5s2bJ6fTKUnq16+f/vKXv0iSvvjiC/Xp00d79uyRj4+Pli9fflE/7dq1U2ZmpiIiIlwhb9u2bUpMTLyxXwDqjJiYGDVt2lStWrWSt7e3Bg0aJEny9/dXQUGBSkpKtHXrVo0YMUJ2u10PPvigDh8+7OaqAaBuIMgBwDWKiIjQ4cOH1bNnT/3iF79QgwYNFBERodatW+tPf/qTYmJiFBAQoKCgIN11110XnV9cXKzY2FjZbDZFRUVp7ty5kqT58+crOztbAwYMUL169bRx40ZJZ5ZzFhQUXNTP4MGDJZ35R3VISIjrH9wNGjRQYWHhjfsCUGd4eXm5PterV8+1Xa9ePVVWVur06dPy8fHRP//5T3l5eckwDC1YsOCaruF0OrVu3bpqrRuXx5NzgdqBe+QA4BrdcccdqqiocG3v37/f9fm+++7Tfffdd9E5JSUlrs+tW7fW9u3bLzqmZcuWWrZsmQoKChQbG+taIunh4aGysrKLjj/3H9UX/oO7srLyOkaGuqhp06YqLi6+rnObNWsmX19fPf/88+rcubNSUlKUm5t7TX04nU5lZ2dr4MCB11VDbVdQUKABAwYoJCREO3fuVKdOnfTWW28pMzNTU6ZMUWVlpbp376433nhDXl5e2rBhwyXbfX19NXbsWH3yySeaOHGiRo0a5e6hAfiJmJEDAKAO+9nPfia73S4vLy9FR0fryy+/1PDhw3XixAl99913ioqKUnBwsL799lv997//lXTmvrrMzExFRUUpJiZGf/3rX7V06VI1bNhQaWlp+uSTT9SzZ08FBQVpxIgRrh8ysrKyFBYWpoCAAPXo0UNFRUVKTEzUsmXLZLfbtWzZMnd+FTetffv2acKECcrNzVWzZs00Z84cxcfHa9myZdq9e7cqKyv1xhtvqLy8/JLtZzVo0EBbtmwhxAG1BEEOAIA67pVXXtHJkye1dOlSlZeXq1mzZvr+++/1zTffKC0tTTk5OUpJSdGcOXMkSb/85S8VEhKiTZs26YUXXtAbb7yhhIQElZeX67HHHtOsWbO0fv167dixQw6HQ3PmzNHJkyc1cuRIvfLKK9q1a5fWr1+vxo0ba+bMmRo5cqScTqdGjhzp5m/i5tSuXTuFh4dLksaMGaMNGzaoQ4cO6tSpkyQpLi5Omzdv1r59+y7ZfhbfL1C7sLQSAG4yvr6+ysvLc22ffWH4uc69Zy4+Pl7x8fGX3AdcrQvDwgsvvKC8vDz17dtXJ0+e1FdffaXIyEjX8VWFgm3btmnv3r2uvk6ePKmePXtq3759at26tbp37y7pzLJMXB3DMK7qONM0L7u/cePG1VEOgJsEM3IAAOCisNC0aVN169bN9TCS3/zmN/rkk09c+6sKBaZpqm/fvnI6nXI6ndq7d68WLVok0zSvOpDgfAcOHFBmZqYkKTU1VX369FFBQYG+/PJLSdLbb7+tqKgode7c+ZLtAGonghwAALXMH//4R3Xu3Fl9+/bVvffeq6SkJDmdToWGhspms2nIkCE6fvy4pDMPGxkyZIgOHDigqKgoHT9+XKmpqfrVr36lHTt2yM/PT6+99ppM09SePXuueO3Q0FB99tlnrjBRWlqq/fv3q3Pnzjp06JCysrIknXl6a2Vl5U962Epd0aVLFy1ZskQ2m03Hjh3T5MmTtXjxYo0YMUL+/v6qV6+eEhIS1KBBg0u2A6idCHIAANQi2dnZWr58uXbu3KkVK1YoOztbknT//ffrpZdeUm5urvz9/fXcc8+52qdOnaouXbqopKREHTt21LFjx7Rx40a99tpratGihVJSUpSfn6+tW7de8fqtWrVSSkqK7r33XtlsNoWGhurzzz/XLbfcomXLlmnSpEkKCAhQ3759VV5erpiYGO3du5eHnVxGvXr1lJycrNzcXC1fvlyNGjXSHXfcoZ07d2r37t168803XU+uraq9oKBALVu2dOcwAFQz40rrqd3J4XCYZ/8CAgAAVzZv3jwdP37cFdQef/xxeXt7a9GiRTpw4IAkKT8/XyNGjNDGjRvl7++vzZs3KzY2VqtXrz6v/ezxubm5uu+++867dxM14+zrSPjugbrLMIwc0zQdF7YzIwcAQC1SHT/Qcj/bzePChx8BwFkEOQAAapFevXpp7dq1Ki8vV0lJiT788EM1btxYzZs3V0ZGhqT//xAMb29vNW/eXAcPHlReXp6r3cfHR97e3tqyZYskaenSpe4cEgDgEnj9AAAAtUj37t01ePBgBQQEqH379nI4HPL29taSJUuUkJCg0tJS3XrrrVq8eLEkVdm+ePFijR07Vo0aNVL//v3dOSQAwCVwjxwAALVMSUmJmjRpotLSUkVGRmrBggUKCgpyd1kAgOtQ1T1yzMgBAFDLTJgwQXv37lV5ebni4uIIcQBQCxHkAACoZd555x13lwAAuMF42AkAAAAAWAxBDgAAAAAshiAHoEr33nuvbDab5s6de9XnrFq1Snv37r0h9cybN0+lpaVXPC46Olo8KAkAANRmBDkAl/Tf//5XW7duVW5uriZPnnxV51RWVt4UQQ7XbsaMGUpKSnJ3GVflev8fS09P19atW6943Jo1a/Tiiy9e9phDhw5p+PDh11wDAADVhSAH1BEFBQXy8/NzbSclJWnGjBmKjo7W1KlT1aNHD3Xq1Mn1wuB+/frpu+++k91uV0ZGhpxOp0JDQ2Wz2TRkyBAdP35c0pnZr6eeekpRUVF66aWXtGbNGj3xxBOy2+3Kz88/b3bs6NGj8vX1lSSlpKRo6NChGjBggDp27Kgnn3zSVdtDDz0kh8Ohbt266dlnn5UkzZ8/X4cOHVJMTIxiYmIkSZ988ol69uypoKAgjRgxQiUlJeeNedGiReeF0IULF+rxxx+v5m+29qqsrHR3CZd0PUGusrLyqoPc4MGDNW3atMse06ZNG6WlpV1TDQAAVCeCHABVVlZq+/btmjdvnp577jlJZ2YlbrvtNjmdTkVEROj+++/XSy+9pNzcXPn7+7uOk6TCwkJt2rRJ06dP1+DBgzV79mw5nU7ddtttl72u0+nUsmXLtHv3bi1btkwHDx6UJD3//PPKzs5Wbm6uNm3apNzcXD366KNq06aNNm7cqI0bN+ro0aOaNWuW1q9frx07dsjhcGjOnDnn9T9q1CitWbNGFRUVks684PiBBx6ozq/O0p5//nndfvvt6tOnj/bt2yfp/GD+yiuvaO3atQoJCVFgYKD69Omjb7/9VtKZGby4uDj169dPvr6+WrFihZ588kn5+/trwIABru985syZ6t69u/z8/DRhwgRV9e7Sv//97+rRo4fsdrsefPBBnTp1Sk2aNNH06dMVEBCg0NBQffvtt9q6detFPxbk5+drwIABCg4OVkREhD7//HNJUnx8vB5//HHFxMRo5MiRSk5O1ty5c10/TlQ1tpSUFE2cONHVx6OPPqqwsDDdeuutrvB27g8jl/tRYtGiRerUqZOio6M1fvx4V78AAPxUBDkAGjp0qCQpODhYBQUFF+0vKipSYWGhoqKiJElxcXHavHmza//IkSOv67p33HGHvL291aBBA3Xt2lX/+c9/JEnvvfeegoKCFBgYqD179lxy9mXbtm3au3evwsPDZbfbtWTJEtf5ZzVu3Fi9e/fWBx98oM8//1wVFRXy9/e/rlprm5ycHL377rvauXOnVqxYoaysLNe+s8H897//vXr16qVt27Zp586dGjVqlF5++WXXcfn5+frwww+1evVqjRkzRjExMdq9e7caNmyoDz/8UJI0ceJEZWVlKS8vT2VlZfrggw8uquXf//63li1bps8++0xOp1MeHh5aunSpfvjhB4WGhmrXrl2KjIzUwoULFRYWdtGPBRMmTNCrr76qnJwcJSUl6eGHH3b1vX//fq1fv17Lly9XQkKCJk+e7Ppx4nJjO9fhw4e1ZcsWffDBB1XO1F3qR4lDhw7pj3/8o7Zt26Z//vOfroAJAEB14D1yQB3h6emp06dPu7bLy8tdn728vCRJHh4e17WcrnHjxld13XOvee51z732119/raSkJGVlZal58+aKj4+/6DxJMk1Tffv2VWpq6mVrGzdunF544QV17tyZ2bhzZGRkaMiQIWrUqJGkM8sJzzo3mH/zzTcaOXKkDh8+rJMnT6pDhw6ufXfeeafq168vf39/nTp1SgMGDJAk+fv7u34Q2Lhxo15++WWVlpbq2LFj6tatmwYNGnReLRs2bFBOTo66d+8uSSorK9PPf/5z3XLLLYqNjZV05keGf/7znxeNo6SkRFu3btWIESNcbSdOnHB9HjFihDw8PC75HVxubOe6++67Va9ePXXt2tU1a3ehsz9KSHL9KHH06FFFRUWpRYsWrlr2799/yfMBALhWzMgBdcQvfvELfffdd/rf//6nEydOXHJmpCre3t5q3ry56/65t99+2zU7d6GmTZuquLjYte3r66ucnBxJuqp7ir7//ns1btxY3t7e+vbbb/XRRx9dsu/Q0FB99tln+vLLLyVJpaWll/xHckhIiA4ePKh33nlH995771WOuG4wDOOS7ecG80mTJmnixInavXu3/vrXv17yB4B69eqpfv36rv7q1aunyspKlZeX6+GHH1ZaWpp2796t8ePHVxnK4+Li5HQ65XQ6tW/fPs2YMeO8Pqv6keH06dPy8fFxnet0OvXvf//7kmO50OXGdq5zf3CoamnopX6UqOpYAACqA0EOqCPq16+vxMREhYSEKDY2Vp07d76m85csWaInnnhCNptNTqdTiYmJlzxu1KhRmj17tgIDA5Wfn68pU6bojTfeUFhYmI4ePXrF6wQEBCgwMFDdunXT2LFjFR4e7to3YcIE3XnnnYqJiVGrVq2UkpLiekVCaGholUvX7rnnHoWHh6t58+bXNObaLDIyUitXrlRZWZmKi4u1du3aSx5XVFSktm3bSjrz/8C1OBuMWrZsqZKSkiqD/B133KG0tDR99913kqRjx45dtEz2XOcG+mbNmqlDhw56//33JZ0JWrt27brieT91bFejR48e2rRpk44fP67KykotX7682q8BAKi7WFoJ1CGPPvqoHn300Sr3t2zZ0rUkztfXV3l5ea59drtd27Ztu+ic9PT087bDw8MvuqctNzfX9XnWrFmSzjxEIj4+3tV+7gxhSkrKJeubNGmSJk2a5Nru3bv3efd2VVXTli1brvoVCnVFUFCQRo4cKbvdrvbt2ysiIuKSx82YMUMjRoxQ27ZtFRoaqq+//vqqr+Hj46Px48fL399fvr6+rqWTF+ratatmzZqlfv366fTp06pfv75ee+21KvsdNWqUxo8fr/nz5ystLU1Lly7VQw89pFmzZqmiokKjRo1SQEDARecNGjRIw4cP1+rVq/Xqq6/+pLFdjbZt2+qpp55SSEiI2rRpo65du7qWXwIA8FMZN/PSD4fDYfJSXwDXq7CwUD169FBAQIBrxgaoSSUlJWrSpIkqKys1ZMgQjR07VkOGDHF3WQAACzEMI8c0TceF7czIAai1fHx8eLgE3GrGjBlav369ysvL1a9fP919993uLgkAUEsQ5AAAuEGSkpLcXQIAoJbiYScAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwAAAAAWQ5ADAAAAAIshyAEAAACAxRDkAAAAAMBiCHIAAAAAYDEEOQAAAACwGIIcAAAAAFgMQQ4AAAAALIYgBwCSMjIy1K1bN9ntdpWVlVV5XHR0tLKzs2uwMgAAgIsR5ABA0tKlSzVlyhQ5nU41bNjQ3eUAAABcFkEOQK1z9913Kzg4WN26ddOCBQv03nvv6fHHH5ckvfLKK7r11lslSfn5+erVq5f+9re/6b333tPMmTM1evRopaenKzY21tXfxIkTlZKS4o6hAAAAXJKnuwsAgOr25ptvqkWLFiorK1P37t318ccfa/bs2ZLOLKH82c9+pv/7v//Tli1bFBERoXHjxmnLli2KjY3V8OHDlZ6e7t4BAAAAXAFBDkCtM3/+fK1cuVKSdPDgQR08eFAlJSUqLi7WwYMHdd9992nz5s3KyMjQ0KFD3VwtAADAtWNpJYBaJT09XevXr1dmZqZ27dqlwMBAlZeXq2fPnlq8eLFuv/12RUREKCMjQ5mZmQoPD7+oD09PT50+fdq1XV5eXpNDAAAAuCKCHIBapaioSM2bN1ejRo30+eefa9u2bZKkyMhIJSUlKTIyUoGBgdq4caO8vLzk7e19UR/t27fX3r17deLECRUVFWnDhg01PQwAAIDLYmklgFplwIABSk5Ols1m0+23367Q0FBJUkREhA4ePKjIyEj9P/buPK7KMv//+OsGF1QMzW2y/AaWCQKHVUNly31cSFNTQxMdF9R0smmxycxsmzEyv1TmaCNYmlG4tIyWoqKimIIiomK4kJX+XHIgSSiW+/eHeb7ivsHx4Pv5ePjw3Nt1f67zmEdz3l73fV2Ojo40a9YMd3f3i7bRrFkzHn30USwWCy1atMDPz68yuyAiIiJyRYZpmrau4ZICAwNNrdckIiIiIiK3K8Mw0k3TDDx/vx6tFBERERERsTMKciIiIiIiInZGQU5ERERERMTOKMiJiIiIiIjYGQU5ERERERERO6MgJyIiIiIiYmcU5EREREREROyMgpyIiIiIiIidUZATERERERGxMwpyIiIiIiIidkZBTkRERERExM4oyImIiMhFzZ49mw8//NDWZYiIyEVUs3UBIiIicmuKjo62dQkiInIJGpETERGpAnJzc3F3d2fEiBF4eXkRGRlJUlIS7du3p0WLFmzZsoWTJ0/Su3dvLBYLQUFBZGZmUlZWhqurK3l5eda27r//fo4ePcrUqVOJiYkBYP/+/XTr1o2AgABCQkLIzs62VVdFRAQFORERkSpj3759/PWvfyUzM5Ps7Gw+/vhjUlJSiImJ4fXXX+ell17Cz8+PzMxMXn/9dR5//HEcHBx4+OGHWbp0KQDffvstrq6uNGnSpFzbo0aN4p133iE9PZ2YmBjGjh1riy6KiMgf9GiliIhIFeHm5oa3tzcAnp6edOzYEcMw8Pb2Jjc3l++//57FixcD0KFDB37++Wfy8/MZMGAA06ZNY9iwYXzyyScMGDAAgKKiIjIzMykoKGDTpk3079/feq/ffvvtojVERUXRs2dP+vXrV8G9FRG5vSnIiYiIVBE1a9a0fnZwcLBuOzg4UFJSQrVqF/7fvmEYtG3bln379nH8+HGWLVvG5MmTgTNBbtOmTZSVlVGvXj0yMjIqpyMiInJFerRSRETkNhEaGsrChQsBSE5OpmHDhtxxxx0YhkGfPn146qmn8PDwoEGDBgAkJSXx888/ExoaioODAxEREXh5eeHl5cX06dMBME2TJ554glatWtGjRw+OHTtmvd+0adNo3bo1Xl5ejBo1CtM02b9/P/7+/tZzcnJyCAgIqMRvQUSkalCQExERuU1MnTqVtLQ0LBYLkyZNYv78+dZjAwYMYMGCBdbHKgE6depEgwYNyMjI4IUXXmDTpk04Ojry+++/89prr3HkyBGWLl3K3r172blzJ3PnzmXTpk3W65944gm2bt1KVlYWhYWFfPXVV9x33324uLhYR/fi4uKIioqqtO9ARKSq0KOVIiIiVYCrqytZWVnW7fj4+Ise+/zzzy96fWBgIKZpltv35JNPkpSUBJwZOZs+fTrDhw8HYMiQIWzdupX169czaNAgHB0dadq0KR06dLBev3btWqZPn87p06c5efIknp6e9OrVixEjRhAXF8eMGTNISEhgy5YtN+U7EBG5nWhETkTkKsXGxuLh4UFkZOQ1X/v6669f933j4+M5fPiwdXvEiBHs3r37utsTuR7nh7xzGYZxwb6ioiLGjh1LYmIiO3fuZOTIkRQVFQHQt29fVqxYwVdffUVAQID1UU4REbl6CnIiIldp1qxZLF++3PqO0bW4mUHugw8+oFWrVtfdnsjVqlu3LqdOnQLOvF+XkJBAaWkpx48fZ/369bRp04bQ0FA++eQTSktLOXLkCGvXrgWwhraGDRtSUFBAYmKitV0nJye6du3KmDFjGDZsWOV3TESkClCQExG5CtHR0Rw4cICIiAj++c9/0q5dO/z8/GjXrh179+4FzgSuRx55hG7dutGiRQueffZZACZNmkRhYSG+vr7W0bzevXsTEBCAp6cnc+bMAaC0tJSoqCi8vLzw9vbm7bffJjExkbS0NCIjI/H19aWwsJDw8HDS0tIA+Prrr/H398fHx4eOHTva4JuRqqxBgwa0b98eLy8vUlNTsVgs+Pj40KFDB6ZPn86f/vQn+vTpQ4sWLfD29mbMmDGEhYUBUK9ePUaOHIm3tze9e/emdevW5dqOjIzEMAy6dOlii66JiNg943KPSthaYGCgefbHioiIrbm6upKWlkaNGjWoXbs21apVIykpiffff5/FixcTHx/PtGnT2L59OzVr1qRly5akpKTQrFkznJ2dKSgosLZ18uRJ7rzzTgoLC2ndujXr1q0jNzeXSZMmsWrVKgDy8vKoV68e4eHhxMTEEBgYCGDdvvfee/H392f9+vW4ublZ2xSxBzExMeTn5/PKK6/YuhQRkVuaYRjppmkGnr9fk52IiFyj/Px8hg4dSk5ODoZhUFxcbD3WsWNHXFxcAGjVqhXff/89zZo1u6CN2NhYli5dCsAPP/xATk4OLVu25MCBA4wfP54ePXpccaRi8+bNhIaG4ubmBqAQJ3ajT58+7N+/nzVr1ti6FBERu6VHK0VErtGLL77IQw89RFZWFl9++aX1XSAovyCzo6MjJSUlF1yfnJxMUlISqamp7NixAz8/P4qKiqhfvz47duwgPDyc9957jxEjRly2DtM0LzrJhMitbunSpWRmZtKwYUNblyIiYrduSpAzDKObYRh7DcPYZxjGpIscNwzDiP3jeKZhGP4Xa0dExB7k5+dz9913A+WneL+c6tWrW0fu8vPzqV+/PrVr1yY7O5vNmzcDcOLECcrKyujbty+vvPIK27ZtA8pPOHGutm3bsm7dOg4ePAiceVxTREREbg83bwiyEAAAIABJREFU/GilYRiOwHtAZ+BHYKthGF+Ypnnu3Nh/Blr88edB4P0//hYRsTvPPvssQ4cOZcaMGeXWzLqcUaNGYbFY8Pf3Z968ecyePRuLxULLli0JCgoC4KeffmLYsGGUlZUB8MYbbwAQFRVFdHQ0tWrVIjU11dpmo0aNmDNnDo888ghlZWU0btzY+n6diIiIVG03PNmJYRhtgammaXb9Y/t5ANM03zjnnH8ByaZpLvpjey8Qbprmkcu1rclORERERESuT15eHh9//DFjx469oXa6d+/Oxx9/TF5eHj179iQrK+uCc86fmEtunktNdnIzHq28G/jhnO0f/9h3recAYBjGKMMw0gzDSDt+/PhNKE9ERERE5PaTl5fHrFmzrvp80zStT4Wcu718+XLq1atXESXKDbgZQe5ib9qfP8x3Neec2Wmac0zTDDRNM7BRo0Y3XJyIiIiI2Lfk5GR69uxp6zLszqRJk9i/fz++vr4888wzvPnmm7Ru3RqLxcJLL70EQG5uLh4eHowdOxZ/f382bNhQbvuHH37A1dWVEydOAFBSUsLQoUOxWCz069eP06dPX3DflStX0rZtW/z9/enfv3+55Xfk5rkZQe5H4Ny5te8BDl/HOSIiIiIicpP84x//4L777iMjI4POnTuTk5PDli1byMjIID09nfXr1wOwd+9eHn/8cbZv38699957wfa59u7dy6hRo8jMzOSOO+64YMTvxIkTvPrqqyQlJbFt2zYCAwOZMWNGpfX5dnIzgtxWoIVhGG6GYdQABgJfnHfOF8Djf8xeGQTkX+n9OBERERG59Xz44YdYLBZ8fHwYMmQI33//PR07dsRisdCxY0cOHToEnJmoacyYMTz00EM0b96cdevWMXz4cDw8PIiKirK2d6nRm6+//hp3d3eCg4NZsmQJAGVlZbRo0YKzr9+UlZVx//33W0eL5NJWrlzJypUr8fPzw9/fn+zsbHJycgC49957rRNvXWz7XM2aNaN9+/YADB48mJSUlHLHN2/ezO7du2nfvj2+vr7Mnz+f77//voJ6dXu74VkrTdMsMQzjCeAbwBGYZ5rmLsMwov84PhtYDnQH9gGngWE3el8RERERqVy7du3itddeY+PGjTRs2JCTJ08ydOhQHn/8cYYOHcq8efOYMGECy5YtA+C///0va9as4YsvvqBXr15s3LiRDz74gNatW5ORkcE999xjHb2pU6cO//znP5kxYwbPPvssI0eOZM2aNdx///0MGDAAAAcHBwYPHszChQt58sknSUpKwsfHR2sSXgXTNHn++ecZPXp0uf25ubnUqVOn3L7zt891/vql52+bpknnzp1ZtGjRDVYsV3JT1pEzTXO5aZoPmKZ5n2mar/2xb/YfIQ7zjHF/HPc2TVNTUYqIiIjYmTVr1tCvXz9rcLrzzjtJTU3lscceA2DIkCHlRmh69eqFYRh4e3vTpEkTvL29cXBwwNPTk9zc3EuO3mRnZ+Pm5kaLFi0wDIPBgwdb2xw+fDgffvghAPPmzWPYMI0PXMq565B27dqVefPmWUc8f/rpJ44dO3bNbR46dMi6FM6iRYsIDg4udzwoKIiNGzeyb98+AE6fPs133313I92QS7jhETkRERERuT2YpnnBCMz5zj1es2ZN4MxI2tnPZ7dLSkpwdHS86OhNRkbGJe/TrFkzmjRpwpo1a/j2229ZuHDh9XanymvQoAHt27fHy8uLP//5zzz22GO0bdsWAGdnZxYsWICjo+M1tenh4cH8+fMZPXo0LVq0YMyYMeWON2rUiPj4eAYNGsRvv/0GwKuvvsoDDzxwczolVgpyIiIiInJVOnbsSJ8+fZg4cSINGjTg5MmTtGvXjk8++YQhQ4awcOHCC0ZoLicoKIhx48axb98+7r//fk6fPs2PP/6Iu7s7Bw8eZP/+/dx3330XBL0RI0YwePBghgwZcs1B5Hbz8ccfl9v+61//esE5564L5+rqesE6cbm5uQA0bNiQ3bt3X/Q+ycnJ1s8dOnRg69at11mxXK2b8miliIiIiFR9np6evPDCC4SFheHj48NTTz1FbGwscXFxWCwWPvroI/73f//3qts7d/TGYrEQFBREdnY2Tk5OzJkzhx49ehAcHHzBzIkREREUFBTosUq5rRmmedHl3G4JgYGBZlqaXqcTERERkf+TlpbGxIkT2bBhg61LEalwhmGkm6YZeP5+PVopIiIiInbjH//4B++//77ejZPbnkbkREREREREblGXGpHTO3IiIiIiIiJ2RkFORERERETEzijIiYhIlRYbG4uHhweRkZG2LkVEROSm0WQnIiJSpc2aNYsVK1bg5uZm61JERERuGo3IiYhIlRUdHc2BAweIiIjAxcWFmJgY6zEvLy9yc3PJzc3Fw8ODkSNH4unpSZcuXSgsLLRh1SIiIlemICciIlXW7Nmzadq0KWvXrmXixImXPC8nJ4dx48axa9cu6tWrx+LFiyuxShERkWunICciIrc9Nzc3fH19AQgICCA3N9e2BYmIiFyBgpyIiNwWqlWrRllZmXW7qKjI+rlmzZrWz46OjpSUlFRqbSIiItdKQU5ERG4Lrq6ubNu2DYBt27Zx8OBBG1ckIiJy/RTkROxEbm4uXl5eF+yfMmUKSUlJAISHh5OWlnZN7Z57vUhV1rdvX06ePImvry/vv/8+DzzwgK1LEhERuW5afkDEzk2bNu26ry0tLb2h60Xswbnvu61cufKi52RlZVk/P/300xVdkoiIyA3TiJyIHSktLb1givSoqCgSExMvOHfMmDEEBgbi6enJSy+9ZN3v6urKtGnTCA4O5rPPPit3/erVq/Hz88Pb25vhw4fz22+/Wa85ceIEAGlpaYSHhwOwbt06fH198fX1xc/Pj1OnTlXwNyAiIiIioCAnYleuZYr01157jbS0NDIzM1m3bh2ZmZnWY05OTqSkpDBw4EDrvqKiIqKiokhISGDnzp2UlJTw/vvvX7aemJgY3nvvPTIyMtiwYQO1atW68U6KVJDY2Fg8PDyoX78+//jHPwCYOnVqubXlRERE7IWCnIgduZYp0j/99FP8/f3x8/Nj165d7N6923pswIABF5y/d+9e3NzcrO8NDR06lPXr11+2nvbt2/PUU08RGxtLXl4e1arpaW25dc2aNYvly5fz3//+l0mTJtm6HBERkRuiICdiR652ivSDBw8SExPD6tWryczMpEePHuWmWq9Tp84F15imecn7njtt+7ntTJo0iQ8++IDCwkKCgoLIzs6+5j6JVIbo6GgOHDhAREQEb7/9Nk888cQF54SHhzNx4kRCQ0Px8PBg69atPPLII7Ro0YLJkyfboGoREZFLU5ATqYJ++eUX6tSpg4uLC0ePHmXFihVXvMbd3Z3c3Fz27dsHwEcffURYWBhw5h259PR0gHKPc+7fvx9vb2+ee+45AgMDFeRughEjRpQbPb0Rzs7ON6WdqmD27Nk0bdqUtWvXUr9+/UueV6NGDdavX090dDQPP/ww7733HllZWcTHx/Pzzz9XYsUiIiKXpyAnUgX5+Pjg5+eHp6cnw4cPp3379le8xsnJibi4OPr374+3tzcODg5ER0cD8NJLL/HXv/6VkJAQHB0drdfMnDkTLy8vfHx8qFWrFn/+858rrE+3iw8++IBWrVrZuozbVkREBADe3t54enpy1113UbNmTZo3b84PP/xg4+pERET+j15oEbETrq6uV5wiPTk52fo5Pj7+ou2c/17dued17NiR7du3X3BNSEgI33333QX733nnncsXLZf166+/8uijj/Ljjz9SWlrKiy++yPvvv09MTAyBgYE4Ozszbtw4kpKSqF+/Pq+//jrPPvsshw4dYubMmURERBAfH8/SpUv57bffOHjwII899li5WUrPevPNN/n000/57bff6NOnDy+//LINenzrO/v4soODQ7lHmR0cHC75KLOIiIgtaERORMRGvv76a5o2bcqOHTvIysqiW7du5Y7/+uuvhIeHk56eTt26dZk8eTKrVq1i6dKlTJkyxXreli1bWLhwIRkZGXz22WcXLAq/cuVKcnJy2LJlCxkZGaSnp19xIhsRERG5tSnIiYjYiLe3N0lJSTz33HNs2LABFxeXcsdr1KhhDXfe3t6EhYVRvXp1vL29y42sdu7cmQYNGlCrVi0eeeQRUlJSyrWzcuVKVq5ciZ+fH/7+/mRnZ5OTk1Ph/RMREZGKo0crRURs5IEHHiA9PZ3ly5fz/PPP06VLl3LHq1evjmEYQPlH/c5/zO/sOZfaNk2T559/ntGjR1dEN+zG2fAbFRVFVFQUcGYdubPOfTQ5PDzcuvD9+cdERERuBRqRExGxkcOHD1O7dm0GDx7M008/zbZt266rnVWrVnHy5EkKCwtZtmzZBZPbdO3alXnz5lFQUADATz/9xLFjx264fhEREbEdjciJiNjIzp07eeaZZ3BwcKB69eq8//77F53E5kqCg4MZMmQI+/bt47HHHiMwMLDc8S5durBnzx7atm0LnFmWYMGCBTRu3Pim9ENEREQqn3G5RYBtLTAw0Dz/pX0REfk/8fHxpKWl8e6771b6vXNzc+nZs2e52VQBpkyZQmhoKJ06dbrktVOnTsXZ2fmiwdXZ2dk6eigiInK7Mwwj3TTNwPP3a0RORERuqmnTptm6BBERkSpP78iJiNixqKgom4zGnVVaWsrIkSPx9PSkS5cuFBYWEhUVRWJiIgDLly/H3d2d4OBgJkyYQM+ePa3X7t69m/DwcJo3b05sbOwFbQ8ZMoTPP//cuh0ZGckXX3xR8Z0SERGxAwpyIiJy3XJychg3bhy7du2iXr16LF682HqsqKiI0aNHs2LFClJSUjh+/Hi5a7Ozs/nmm2/YsmULL7/8MsXFxeWOjxgxgri4OADy8/PZtGkT3bt3r/hOiYiI2AEFORERuW5ubm74+voCEBAQUG59u+zsbJo3b46bmxsAgwYNKndtjx49qFmzJg0bNqRx48YcPXq03PGwsDD27dvHsWPHWLRoEX379qVaNb0RICIiAnpHTkREbsDZte0AHB0dKSwstG5faTKt8689d228s4YMGcLChQv55JNPmDdv3k2oWEREpGpQkBMRkQrh7u7OgQMHyM3NxdXVlYSEhGtuIyoqijZt2vCnP/0JT0/PCqhSRETEPunRShERqRC1atVi1qxZdOvWjeDgYJo0aYKLi8s1tdGkSRM8PDwYNmxYBVUpYjvZ2dm0a9cOb29vwsLCOHHihK1LEhE7onXkRESkwhQUFODs7IxpmowbN44WLVowceLEq77+9OnTeHt7s23btmsOgSK3uuzsbGrUqEHz5s15/vnnqVOnDpMnT7Z1WSJyi7nUOnIakRMRkQozd+5cfH198fT0JD8/n9GjR1/1tUlJSbi7uzN+/HiFOKmS3N3dad68OXBmllcnJycbVyQi9kQjciIiIiI29M033/Dkk0+SmppKvXr1bF2OiNxiLjUip8lORERERGykrKyMv/zlL6xdu1YhTkSuiR6tFBEREbGRw4cP4+LiQosWLWxdiojYGQU5ERERERupX78+b731lq3LEBE7pCAnIiIiYiP5+fl88MEHti5DROyQgpyIiIiIjTRt2pTExERbl2HXoqKi9B3KbUlBTkRERETsUmlpqa1LELEZBTkRERERqXC5ubm4u7szdOhQLBYL/fr14/Tp06xevRo/Pz+8vb0ZPnw4v/32G8Al97u6ujJt2jSCg4P57LPPrO2vXr2aPn36WLdXrVrFI488UrmdFKlECnIiIiIiUin27t3LqFGjyMzM5I477mDGjBlERUWRkJDAzp07KSkp4f3336eoqOii+89ycnIiJSWFgQMHWvd16NCBPXv2cPz4cQDi4uIYNmxYpfdRpLIoyImIiIhIpWjWrBnt27cHYPDgwaxevRo3NzceeOABAIYOHcr69evZu3fvRfefNWDAgAvaNgyDIUOGsGDBAvLy8khNTeXPf/5zJfRKxDa0ILiIiIiIVArDMK7qPNM0L3u8Tp06F90/bNgwevXqhZOTE/3796daNf3UlapLI3IiIiIiUikOHTpEamoqAIsWLaJTp07k5uayb98+AD766CPCwsJwd3e/6P4radq0KU2bNuXVV18lKiqqwvohcitQkBMRERGRSuHh4cH8+fOxWCycPHmSiRMnEhcXR//+/fH29sbBwYHo6GicnJwuuv9qREZG0qxZM1q1alXBvRGxLeNKQ9e2FBgYaKalpdm6DBERERG5Qbm5ufTs2ZOsrKwKvc8TTzyBn58ff/nLXyr0PiKVxTCMdNM0A8/frweHRURERKRKCAgIoE6dOrz11lu2LkWkwinIiYiIiEiFc3V1rfDRuPT09AptX+RWonfkRERERERE7IyCnIiIiIiIiJ1RkBMREREREbEzCnIiIiIiIiJ2RkFORERERETEzijIiYiIiIiI2BkFORERERERETujICciIiIiImJnFORERERERETsjIKciIiIiIiInVGQExERERERsTMKciIiIiIiInZGQU5ERERERMTOKMiJiIiIiIjYGQU5ERERERERO6MgJyIiIiIiYmcU5EREREREROyMgpyIiIiIiIidUZATERERERGxMwpyIiIiIiIidkZBTkRERERExM4oyImIyC1l6tSpxMTEMGXKFJKSkmxaS/fu3cnLy7vsOfHx8Rw+fLjCa6ms+4iIiH1QkBMRkVvStGnT6NSpk01rWL58OfXq1bvsOQpyIiJiCwpyIiJic6+99hotW7akU6dO7N27F4CoqCgSExMBmDRpEq1atcJisfD0008D8OWXX/Lggw/i5+dHp06dOHr0KHBmRG/IkCF06NCBFi1aMHfuXACSk5MJDQ2lT58+tGrViujoaMrKygBYtGgR3t7eeHl58dxzz1nrcnV15cSJE+Tm5uLh4cHIkSPx9PSkS5cuFBYWkpiYSFpaGpGRkfj6+lJYWIirqyt///vfadu2LYGBgWzbto2uXbty3333MXv2bGvbb775Jq1bt8ZisfDSSy8BXNN9RETk9qYgJyIiNpWens4nn3zC9u3bWbJkCVu3bi13/OTJkyxdupRdu3aRmZnJ5MmTAQgODmbz5s1s376dgQMHMn36dOs1mZmZ/Oc//yE1NZVp06ZZR7K2bNnCW2+9xc6dO9m/fz9Llizh8OHDPPfcc6xZs4aMjAy2bt3KsmXLLqgzJyeHcePGsWvXLurVq8fixYvp168fgYGBLFy4kIyMDGrVqgVAs2bNSE1NJSQkxBpIN2/ezJQpUwBYuXIlOTk5bNmyhYyMDNLT01m/fv0130dERG5f1WxdgIiI3N42bNhAnz59qF27NgARERHljt9xxx04OTkxYsQIevToQc+ePQH48ccfGTBgAEeOHOH333/Hzc3Nes3DDz9MrVq1qFWrFg899BBbtmyhXr16tGnThubNmwMwaNAgUlJSqF69OuHh4TRq1AiAyMhI1q9fT+/evcvV4ebmhq+vLwABAQHk5uZesk9n++Dt7U1BQQF169albt26ODk5kZeXx8qVK1m5ciV+fn4AFBQUkJOTw//8z/9c031EROT2pRE5EZHLyM3NxcvL66a1N3v2bD788EMAwsPDSUtLu+Cc+Ph4nnjiiZt2T3tgGMYlj1WrVo0tW7bQt29fli1bRrdu3QAYP348TzzxBDt37uRf//oXRUVFl2zv7PbF9pumeVU11qxZ0/rZ0dGRkpKSK57r4OBQ7joHBwdKSkowTZPnn3+ejIwMMjIy2LdvH3/5y1+u+T4iInL7UpATEakg5/8ALykpITo6mscff9xGFd2aQkNDWbp0KYWFhZw6dYovv/yy3PGCggLy8/Pp3r07M2fOJCMjA4D8/HzuvvtuAObPn1/ums8//5yioiJ+/vlnkpOTad26NXDm0cqDBw9SVlZGQkICwcHBPPjgg6xbt44TJ05QWlrKokWLCAsLu+r669aty6lTp66pz127dmXevHkUFBQA8NNPP3Hs2LGbfh8REam69GiliMgVlJaWMnLkSDZt2sTdd9/N559/zt69e4mOjub06dPcd999zJs3j/r16xMeHk67du3YuHEjERERfPnll+W2T506hbOzs3XCjgULFjBhwgR++eUX5s2bR5s2bcrd+/jx40RHR3Po0CEAZs6cSfv27Sv9O6hI/v7+DBgwAF9fX+69915CQkLKHT916hQPP/wwRUVFmKbJ22+/DZyZ1KR///7cfffdBAUFcfDgQes1bdq0oUePHhw6dIgXX3yRpk2b8t1339G2bVsmTZrEzp07rROfODg48MYbb/DQQw9hmibdu3fn4Ycfvur6o6KiiI6OplatWqSmpl7VNV26dGHPnj20bdsWAGdnZxYsWICjo+NV30fvyYmI3N6Mq32kxBYCAwPNiz12JCJSWXJzc7n//vtJS0vD19eXRx99lIiICKZPn84777xDWFgYU6ZM4ZdffmHmzJmEh4fTqlUrZs2aBXDB9tSpU61BLjw83Dqr4vr16xk7dixZWVnEx8eTlpbGu+++y2OPPcbYsWMJDg7m0KFDdO3alT179tjyK7nlnfsdnys5OZmYmBi++uorG1UmIiJy7QzDSDdNM/D8/RqRExG5gvMnn9i/fz95eXnWx++GDh1K//79recPGDCg3PXnb59r0KBBwJnHC3/55ZcLFp9OSkpi9+7d1u1ffvmFU6dOUbdu3RvrlIiIiNg1BTkRkSs4f/KJ88PW+erUqXPZ7XNdalKOs8rKyvQY3TWaOnXqRfeHh4cTHh5eqbWIiIhUFE12IiJyjVxcXKhfvz4bNmwA4KOPPrqmyTHOlZCQAEBKSgouLi64uLiUO96lSxfeffdd6/bZiT5ERETk9qYRORGR6zB//nzrZCfNmzcnLi7uutqpX78+7dq1s052cr7Y2FjGjRuHxWKhpKSE0NBQZs+efaPli4iIiJ3TZCciIiIiVdSyZct44IEHaNWqla1LEZHrdKnJTvRopYiIiIgduJ7F4ZctW1ZuwiQRqTo0IiciIiJyC3jllVdYuHAhzZo1o2HDhgQEBPDVV1+VW4syPDycp556ioKCAho2bEh8fDx33XUXc+fOZc6cOfz+++/cf//9fPTRR2RkZNCzZ0/r+7eLFy/mvvvus3U3ReQaafkBERERkVtUWloaixcvZvv27ZSUlODv709AQAAAeXl5rFu3juLiYsLCwvj8889p1KgRCQkJvPDCC8ybN49HHnmEkSNHAjB58mT+/e9/M378eCIiIujZsyf9+vWzZfdEpAIoyImIiIjYWEpKCg8//LB1qZFevXpZj51di3Lv3r1kZWXRuXNnAEpLS7nrrrsAyMrKYvLkyeTl5VFQUEDXrl0ruQciUtkU5ERERERs7HKvupxdi9I0TTw9PUlNTb3gnKioKJYtW4aPjw/x8fEkJydXVKkicovQZCciIiJVWGxsLB4eHkRGRlZI+2lpaUyYMAGA5ORkNm3aVCH3qeqCg4P58ssvKSoqoqCggP/85z8XnNOyZUuOHz9uDXLFxcXs2rULgFOnTnHXXXdRXFzMwoULrdfUrVuXU6dOVU4nRKRSaURORESkCps1axYrVqzAzc3tprddUlJCYGAggYFn3sFPTk7G2dmZdu3a3fR7VXWtW7cmIiICHx8f7r33XgIDA3FxcSl3To0aNUhMTGTChAnk5+dTUlLCk08+iaenJ6+88goPPvgg9957L97e3tbwNnDgQEaOHElsbCyJiYma7ESkCtGslSIiIlVUdHQ08+bNo2XLlgwePJjPP/+cwsJCatWqRVxcHC1btuTBBx9k3rx5eHp6AhAeHs5bb72Fm5sbw4cP58CBA9SuXZs5c+ZgsViYOnUqhw8fJjc3l4YNGzJq1ChiYmJ49913CQoKwtHRkUaNGvHOO+/g7u5OdHQ0hw4dAmDmzJm0b9/ell/JLa2goABnZ2dOnz5NaGgoc+bMwd/f39ZliYiNadZKERGR28zs2bP5+uuvWbt2LTVq1OBvf/sb1apVIykpib///e8sXryYgQMH8umnn/Lyyy9z5MgRDh8+TEBAAOPHj8fPz49ly5axZs0aHn/8cTIyMgBIT08nJSWFWrVqWd/FcnV1JTo6GmdnZ55++mkAHnvsMSZOnEhwcDCHDh2ia9eu7Nmzx1Zfxy1v1KhR7N69m6KiIoYOHaoQJyKXpSAnIiJyG8jPz2fo0KHk5ORgGAbFxcUAPProo3Tu3JmXX36ZTz/9lP79+wNnZlFcvHgxAB06dODnn38mPz8fgIiICOvsipeTlJRUbjHqX375hVOnTlG3bt2b3b0q4eOPP7Z1CSJiRxTkREREbgMvvvgiDz30EEuXLiU3N5fw8HAA7r77bho0aEBmZiYJCQn861//Ai4+i6JhGMD/zaJ4JWVlZaSmpl5V6BMRkWujWStFRERuA/n5+dx9990AxMfHlzs2cOBApk+fTn5+Pt7e3gCEhoZaZz9MTk6mYcOG3HHHHZe9x/kzJHbp0oV3333Xun320UwREblxCnIiIiK3gWeffZbnn3+e9u3bU1paWu5Yv379+OSTT3j00Uet+6ZOnUpaWhoWi4VJkyYxf/78K96jV69eLF26FF9fXzZs2EBsbKy1jVatWjF79uyb3i8RkduVZq0UERERERG5RV1q1kqNyImIXMGyZcvKTdggIiIiYmsKciIiV6AgJyIiIrcaBTkRqZKmT59ObGwsABMnTqRDhw4ArF69msGDBzNmzBgCAwPx9PTkpZdesl43adIkWrVqhcVi4emnn2bTpk188cUXPPPMM/j6+rJ//372799Pt27dCAgIICQkhOzsbJv0UW4Nubm5eHl5XfG8KVOmkJSUBJxZdPvsqwOurq6cOHECgHbt2l13HfHx8Rw+fPi6rxcREfui5QdEpEoKDQ3lrbfeYsKECaSlpfHbb79RXFxMSkoKISEh9O/fnzvvvJO5DrIhAAAgAElEQVTS0lI6duxIZmYm99xzD0uXLiU7OxvDMMjLy6NevXpERETQs2dP+vXrB0DHjh2ZPXs2LVq04Ntvv2Xs2LGsWbPGxj2WW1lpaSnTpk274nmbNm267nvEx8fj5eVF06ZNr7sNERGxHxqRE5EqKSAggPT0dE6dOkXNmjVp27YtaWlpbNiwgZCQED799FP8/f3x8/Nj165d7N69mzvuuAMnJydGjBjBkiVLqF279gXtFhQUsGnTJvr374+vry+jR4/myJEjNuih3EpKSkoYOnQoFouFfv36cfr0aVxdXZk2bRrBwcF89tlnREVFkZiYeNl2nJ2dgTP/O+vYsSP+/v54e3vz+eefA2dG/zw8PBg5ciSenp506dKFwsJCEhMTSUtLIzIyEl9fXwoLCyu8zyIiYlsKciJSJVWvXh1XV1fi4uJo164dISEhrF27lv3791OrVi1iYmJYvXo1mZmZ9OjRg6KiIqpVq8aWLVvo27cvy5Yto1u3bhe0W1ZWRr169cjIyLD+2bNnjw16KLeSvXv3MmrUKDIzM7njjjuYNWsWAE5OTqSkpDBw4MBras/JyYmlS5eybds21q5dy9/+9jfrAt05OTmMGzeOXbt2Ua9ePRYvXky/fv0IDAxk4cKFZGRkaAFuEZHbgIKciFRZoaGhxMTEEBoaSkhICLNnz8bX15dffvmFOnXq4OLiwtGjR1mxYgVwZhQkPz+f7t27M3PmTOvixecucnzHHXfg5ubGZ599BoBpmuzYscM2HZRbRrNmzWjfvj0AgwcPJiUlBYABAwZcV3umafL3v/8di8VCp06d+Omnnzh69CgAbm5u+Pr6AmdGnnNzc2+8AyIiYncU5ESkygoJCeHIkSO0bduWJk2a4OTkREhICD4+Pvj5+eHp6cnw4cOtP8BPnTpFz549sVgshIWF8fbbbwMwcOBA3nzzTfz8/Ni/fz8LFy7k3//+Nz4+Pnh6elofe5Pbl2EYF92uU6fOdbW3cOFCjh8/Tnp6OhkZGTRp0oSioiIAatasaT3P0dGRkpKS66xaRETsmSY7EZEqq2PHjhQXF1u3v/vuO+vn+Pj4i16zZcuWC/a1b9/+guUHvv7665tTpFQJhw4dIjU1lbZt27Jo0SKCg4PZvn37dbeXn59P48aNqV69OmvXruX777+/4jXnjhyLiEjVpxE5ERGRG+Th4cH8+fOxWCycPHmSMWPG3FB7kZGRpKWlWd97c3d3v+I1UVFRREdHa7ITEZHbhHH25enrutgw7gQSAFcgF3jUNM3/XuS8XOAUUAqUmKYZeDXtBwYGmmfX2REREREREbndGIaRfrH8dKMjcpOA1aZptgBW/7F9KQ+Zpul7tSFOREREbm9paWlMmDDB1mWIiNySbvQduYeB8D8+zweSgedusE0RERERAgMDCQzUv/+KiFzMjY7INTFN8wjAH383vsR5JrDSMIx0wzBGXa5BwzBGGYaRZhhG2vHjx2+wPBEREbkZevfuTUBAAJ6ensyZMwc4s4D5Cy+8gI+PD0FBQdYlEqKiopgwYQLt2rWjefPm1oXQTdPkmWeewcvLC29vbxISEgAYMmRIudlfIyMj+eKLL0hOTqZnz54ATJ06leHDhxMeHk7z5s2JjY21nv/KK6/g7u5O586dGTRoEDExMZXynYiI2NIVg5xhGEmGYWRd5M/D13Cf9qZp+gN/BsYZhhF6qRNN05xjmmagaZqBjRo1uoZbiIiISEWZN28e6enppKWlERsby88//8yvv/5KUFAQO3bsIDQ0lLlz51rPP3LkCCkpKXz11VdMmnTmzYslS5aQkZHBjh07SEpK4plnnuHIkSOMGDGCuLg44MyMnZs2baJ79+4X1JCdnc0333zDli1bePnllykuLiYtLY3Fixezfft2lixZgt6tF5HbxRUfrTRNs9OljhmGcdQwjLtM0zxiGMZdwLFLtHH4j7+PGYaxFGgDrL/OmkVERKSSxcbGsnTpUgB++OEHcnJyqFGjhnXELCAggFWrVlnP7927Nw4ODrRq1co6UpeSksKgQYNwdHSkSZMmhIWFsXXrViIiIhg3bhzHjh1jyZIl9O3bl2rVLvyJ0qNHD2rWrEnNmjVp3LgxR48eJSUlhYcffphatWoB0KtXr4r+KkREbgk3+mjlF8DQPz4PBS5YFdcwjDqGYdQ9+xnoAmTd4H1FRESkkiQnJ5OUlERqaio7duzAz8+PoqIiqlevbl38/PzFyc9duPzsDNmXmyl7yJAhLFy4kLi4OIYNG3bRcy62GPqNzL4tImLPbjTI/QPobBhGDtD5j20Mw2hqGMbyP85pAqQYhrED2AL8xzRNraQrIiJiJ/Lz86lfvz61a9cmOzubzZs3X1c7oaGhJCQkUFpayvHjx1m/fj1t2rQBzrxXN3PmTAA8PT2vus3g4GC+/PJLioqKKCgo4D//+c911SYiYm9uaNZK0zR/BjpeZP9hoPsfnw8APjdyHxEREbGdbt26MXv2bCwWCy1btiQoKOi62unTpw+pqan4+PhgGAbTp0/nT3/6EwBNmjTBw8OD3r17X1ObrVu3JiIiAh8fH+69914CAwNxcXG5rvpEROzJDS0IXtG0ILiIiMjt4fTp03h7e7Nt27ZrDmIFBQU4Oztz+vRpQkNDmTNnDv7+/hVUqYhI5aqoBcFFREREbkhSUhLu7u6MHz/+ukbTRo0aha+vL/7+/vTt21chTkRuCxqRExERERERuUVpRE5ERERERKSKUJATERGRi2rXrp2tSxARkUtQkBMREZGL2rRpk61LEJFb2OzZs/nwww8rrP3c3Fy8vLwqrH17pyAnIiJiZ3r37k1AQACenp7MmTMHAGdnZ5577jkCAgLo1KkTW7ZsITw8nObNm/PFF18AZ34UhYSE4O/vj7+/vzWoTZkyBV9fX3x9fbn77rutC3I7OzsDZxYEDw8Pp1+/fri7uxMZGWldiHv58uW4u7sTHBzMhAkT6NmzZ2V/HSJiI9HR0Tz++OO2LuO2pSAnIiJiZ+bNm0d6ejppaWnExsby888/8+uvvxIeHk56ejp169Zl8uTJrFq1iqVLlzJlyhQAGjduzKpVq9i2bRsJCQlMmDABgGnTppGRkcG6deto0KABTzzxxAX33L59OzNnzmT37t0cOHCAjRs3UlRUxOjRo1mxYgUpKSkcP368Ur8HEbn5FixYQJs2bfD19WX06NGUlpbi7OzMCy+8gI+PD0FBQRw9ehSAqVOnEhMTA0BGRgZBQUFYLBb69OnDf//7X/bv319uFtmcnBwCAgIASE9PJywsjICAALp27cqRI0es+318fGjbti3vvfdeJffevijIiYiI2JnY2FjrD6offviBnJwcatSoQbdu3QDw9vYmLCyM6tWr4+3tTW5uLgDFxcWMHDkSb29v+vfvz+7du61tmqZJZGQkEydOtP7QOlebNm245557cHBwwNfXl9zcXLKzs2nevDlubm4ADBo0qOI7LyIVZs+ePSQkJLBx40YyMjJwdHRk4cKF/PrrrwQFBbFjxw5CQ0OZO3fuBdc+/vjj/POf/yQzMxNvb29efvll7rvvPlxcXMjIyAAgLi6OqKgoiouLGT9+PImJiaSnpzN8+HBeeOEFAIYNG0ZsbCypqamV2nd7VM3WBYiIiMjVS05OJikpidTUVGrXrk14eDhFRUVUr14dwzAAcHBwoGbNmtbPJSUlALz99ts0adKEHTt2UFZWhpOTk7XdqVOncs8991gfqzzf2fYAHB0dKSkp4VZewkhErt3q1atJT0+ndevWABQWFtK4cWNq1KhhfWw6ICCAVatWlbsuPz+fvLw8wsLCABg6dCj9+/cHYMSIEcTFxTFjxgwSEhLYsmULe/fuJSsri86dOwNQWlrKXXfddUE7Q4YMYcWKFZXSd3ukICciImJH8vPzqV+/PrVr1yY7O5vNmzdf07VnR9Xmz59PaWkpAF999RWrVq0iOTn5mmpxd3fnwIED5Obm4urqSkJCwjVdLyK3FtM0GTp0KG+88Ua5/TExMdZ/KDr7DzlXq2/fvrz88st06NCBgIAAGjRowOHDh/H09Lxg1C0vL896H7kyPVopIiJiR7p160ZJSQkWi4UXX3yRoKCgq7527NixzJ8/n6CgIL777jvq1KkDwFtvvcXhw4et78WcfafuSmrVqsWsWbPo1q0bwcHBNGnSBBcXl+vql4jYXseOHUlMTOTYsWMAnDx5ku+///6K17m4uFC/fn02bNgAwEcffWQdVXNycqJr166MGTPGOuLfsmVLjh8/bg1yxcXF7Nq1i3r16uHi4kJKSgoACxcuvOl9rEqMW/mxiMDAQDMtLc3WZYiIiMglFBQU4OzsjGmajBs3jhYtWjBx4kRblyUi1ykhIYE33niDsrIyqlevznvvvUenTp0oKCgAIDExka+++or4+HimTp1K3bp1+dvf/kZGRgbR0dGcPn2a5s2bExcXR/369QHYvHkzffv25dChQzg6OgJnJkeZMGEC+fn5lJSU8OSTTzJy5EjrO3O1a9ema9euJCYmkpWVZbPv41ZgGEa6aZqBF+xXkBMREZHr9fbbbzN//nx+//13/Pz8mDt3LrVr17Z1WSJSCcaPH4+/v/8l3609KyYmhvz8fF555ZVKqqxqUZATEREREZGb4sUXX+Sbb75hxYoVNGjQ4JLn9enTh/3797NmzRoaNmxYiRVWHQpyIiIiIiIiduZSQU6TnYiIiIiIiNgZBTkRERERERE7oyAnIiIiIiJiZxTkRERERERE7IyCnIiIiIiIiJ1RkBMREREREbEzCnIiIiIiIiJ2RkFORERERETEzijIiYiIiIiI2BkFORERERERETujICciIiIiImJnFORERERERETsjIKciIiIiIiInVGQExERERERsTMKciIiIiIiInZGQU5ERERERMTOKMiJiIiIiIjYGQU5ERERERERO6MgJyIiIiIiYmcU5EREREREROyMgpyIiIiIiIidUZATERERERGxMwpyIiIiIiIidkZBTkRERERExM4oyImIiIiIiNgZBTkRERERERE7oyAnIiIiIiJiZxTkRERERERE7IyCnIiIiIiIiJ1RkBMREREREbEzCnIiIiIiIiJ2RkFORMQORUVFkZiYCEB4eDhpaWnX1U5ycjKbNm26maWJiIhIJVCQExG5jSnIiYiI2CcFORGRCrRgwQLatGmDr68vo0eP5ttvv8VisVBUVMSvv/6Kp6cnWVlZlJaW8vTTT+Pt7Y3FYuGdd94BID09nbCwMAICAujatStHjhy57P1WrlxJ27Zt8ff3p3///hQUFADg6urKSy+9hL+/P97e3mRnZ5Obm8vs2bN5++238fX1ZcOGDRX+fYiIiMjNoSAnIlJB9uzZQ0JCAhs3biQjIwNHR0f27t1LREQEkydP5tlnn2Xw4MF4eXkxZ84cDh48yPbt28nMzCQyMpLi4mLGjx9PYmIi6enpDB8+nBdeeOGS9ztx4gSvvvoqSUlJbNu2jcDAQGbMmGE93rBhQ7Zt28aYMWOIiYnB1dWV6OhoJk6cSEZGBiEhIZXxtYiIiMhNUM3WBYiIVFWrV68mPT2d1q1bA1BYWEjjxo2ZMmUKrVu3xsnJidjYWACSkpKIjo6mWrUz/1m+8847ycrKIisri86dOwNQWlrKXXfddcn7bd68md27d9O+fXsAfv/9d9q2bWs9/sgjjwAQEBDAkiVLbn6HRUREpNIoyImIVBDTNBk6dChvvPFGuf3/7//9PwoKCiguLqaoqIg6depgmiaGYVxwvaenJ6mpqVd9v86dO7No0aKLHq9ZsyYAjo6OlJSUXEePRERE5FahRytFRCpIx44dSUxM5NixYwCcPHmS77//nlGjRvHKK68QGRnJc889B0CXLl2YPXu2NWCdPHmSli1bcvz4cWuQKy4uZteuXZe8X1BQEBs3bmTfvn0AnD59mu++++6yNdatW5dTp07dcF9FRESkcinIiYhUkFatWvHqq6/SpUsXLBYLnTt3Zv78+VSrVo3HHnuMSZMmsXXrVtasWcOIESP4n//5HywWCz4+Pnz88cfUqFGDxMREnnvuOXx8fPD19b3sDJONGjUiPj6eQYMGYbFYCAoKIjs7+7I19urVi6VLl2qyExERETtjmKZp6xouKTAw0LzetZFERERERETsnWEY6aZpBp6/XyNyIiIiIiIidkZBTkRERERExM4oyImIiIiIiNgZBTkRERERERE7oyAnIiIiIiJiZxTkRERERERE7IyCnIiIiIiIiJ1RkBMREREREbEzCnIiIiIiIiJ2RkFORERERETEzijIiYiIiIiI2BkFObmkqVOnEhMTc8njx48f58EHH8TPz48NGzbQvXt38vLyLtvmlClTSEpKAmDmzJmcPn36inWEh4eTlpZ2bcWLiIiIiFRh1WxdgNiv1atX4+7uzvz58wEICQm54jXTpk2zfp45cyaDBw+mdu3aFVajiIiIiEhVpBE5Kee1116jZcuWdOrUib179wKwf/9+unXrRkBAACEhIWRnZ5ORkcGzzz7L8uXL8fX1pbCwEFdXV06cOEFubi4eHh6MHDkST09PunTpQmFhIQBRUVEkJiYSGxvL4cOHeeihh3jooYcAWLlyJW3btsXf35/+/ftTUFBQrrZ///vfTJw40bo9d+5cnnrqqUr6ZkREREREbh0KcmKVnp7OJ598wvbt21myZAlbt24FYNSoUbzzzjukp6cTExPD2LFj8fX1Zdq0aQwYMICMjAxq1apVrq2cnBzGjRvHrl27qFevHosXLy53fMKECTRt2pS1a9eydu1aTpw4wauvvkpSUhLbtm0jMDCQGTNmlLtm4MCBfPHFFxQXFwMQFxfHsGHDKvAbkf/f3r0Ha1nW/QL/XoN5SA3YeXp938aw8RDngYWB2PKUeMw2GSlTo+RGMqUam3FqxmlvLTvokH9kpdk2sCLjFSFT2R5QGxG1gFqYBwQx3DmW4jZF5aDAvf9wuV4PHJatw8O91ucz46znvp/rua/fw8XFrK/X9dwPAAA7JlsrabNgwYKMHz++bavjqaeemvXr1+f+++/PhAkT2tpt2LBhu9caMGBAhg8fniQZOXJkVq1atc32Dz74YB599NGMHTs2SfLaa69lzJgxb2uz++6755hjjsktt9ySj370o3n99dczZMiQ9/IWAQCgRxDkeJtSytuON2/enH79+qWlpeU9XWeXXXZpe9ynT5+2rZVbU1VVjjvuuFx//fXbbDd58uR897vfzaGHHmo1DgCAXsvWSto0Nzdn7ty5WbduXV5++eXcfPPNef/7358BAwbkhhtuSPJG4Fq6dGmn9Lfnnnvm5ZdfTpKMHj06CxcuzBNPPJEkWbt2bZYvX/6u13zsYx/L3/72t/z617/OxIkTO6UOAACoG0GONiNGjMjpp5+e4cOH57TTTmu7C+XMmTNz7bXXZtiwYRk0aFBuuummTulvypQpOfHEE3P00Udn7733zowZMzJx4sQMHTo0o0ePzrJly7b4us9+9rMZO3Zs+vfv3yl1AABA3ZSqqhpdw1Y1NTVVvj+MdzrllFNywQUX5Nhjj210KQAA0KVKKUuqqmp653krctTGiy++mIMPPji77babEAcAQK/mZifURr9+/bb4uTkAAOhtrMgBAADUjCAHPcwee+yRJHnmmWfymc98JkkyY8aMTJ069V+63u9///uccsopnVYfAAAdJ8hBD7X//vtn9uzZjS4DAIAuIMhBD7Vq1aoMHjz4XedvvfXWjBkzJs8//3zuuOOOjBkzJiNGjMiECRPyyiuvJEluu+22HHrooTniiCMyZ86c7i4dAIDtEOSgF5k7d26+//3vZ968eUmSSy+9NPPnz8+f/vSnNDU15Yorrsj69etzzjnn5Oabb86CBQvyj3/8o8FVAwDwToIc9BL33HNPLrvsstx6663p379/HnzwwTz66KMZO3Zshg8fnuuuuy5PPfVUli1blgEDBuSggw5KKSWf//znG106QI+wtZ0SAP8KXz8AvcSBBx6YJ598MsuXL09TU1Oqqspxxx2X66+//m3tWlpaUkppUJUAbMnGjRuz005+bQP+ixU56CUOOOCAzJkzJ2eeeWYeeeSRjB49OgsXLswTTzyRJFm7dm2WL1+eQw89NH/961+zcuXKJHlX0APgX7dp06acc845GTRoUMaNG5d169Zl5cqVOeGEEzJy5Mh8/OMfz7Jly5IkkyZNyte+9rUcffTR+frXv97gyoEdjSAHvcghhxySmTNnZsKECVmzZk1mzJiRiRMnZujQoRk9enSWLVuWXXfdNddcc01OPvnkHHHEETnggAMaXTZAj7FixYqcf/75eeSRR9KvX7/ceOONmTJlSq688sosWbIk06ZNy3nnndfWfvny5Zk/f35+8IMfNLBqYEdUqqpqdA1b1dTUVC1evLjRZQAAdNiqVaty3HHHZcWKFUmSyy67LK+//nq+853v5JBDDmlrt2HDhjz22GOZNGlSjj766Jx11lmNKhnYAZRSllRV1fTO8zZbAwB0k1122aXtcZ8+ffLss8+mX79+aWlp2WL73XffvbtKA2rG1koAgAb5wAc+kAEDBuSGG25IklRVlaVLlza4KqAOBDkAgAaaOXNmrr322gwbNiyDBg3KTTfd1OiSgBrwGTkAAIAd1NY+I2dFDgAAoGYEOQAAgJoR5AAAAGpGkAMAAKgZQQ4AAKBmBDkAAICaEeQAYAdx9dVX5xe/+EWnXvOoo47Klr7KZ8aMGZk6dWqn9gVA99mp0QUAAG8499xzG10CADVhRQ4AutCvfvWrHHbYYRk+fHi++MUvZtOmTdljjz1y0UUXZdiwYRk9enSeffbZJMnFF1+cadOmJUlaWloyevToDB06NOPHj88///nPrFy5MiNGjGi79ooVKzJy5Mgkybe+9a2MGjUqgwcPzpQpU1JV1dtqOPzwwzN48OD88Y9/fFeNq1evzmmnnZZRo0Zl1KhRWbhwYVf+kQDQCQQ5AOgijz32WGbNmpWFCxempaUlffr0ycyZM/Pqq69m9OjRWbp0aZqbm/Ozn/3sXa8988wzc9lll+Whhx7KkCFDcskll+QjH/lI+vbtm5aWliTJ9OnTM2nSpCTJ1KlTs2jRojz88MNZt25dbrnllrZrvfrqq7n//vvzk5/8JGefffa7+vrqV7+aCy64IIsWLcqNN96YyZMnd80fCACdxtZKAOgid911V5YsWZJRo0YlSdatW5d99tknO++8c0455ZQkyciRI3PnnXe+7XUvvfRSXnzxxRx55JFJkrPOOisTJkxIkkyePDnTp0/PFVdckVmzZrWtsN1zzz25/PLLs3bt2rzwwgsZNGhQPvnJTyZJJk6cmCRpbm7OmjVr8uKLL76tv/nz5+fRRx9tO16zZk1efvnl7Lnnnp39RwJAJxHkAKCLVFWVs846K9/73vfedn7atGkppSRJ+vTpk40bN7b7mqeddlouueSSHHPMMRk5cmQ++MEPZv369TnvvPOyePHifOhDH8rFF1+c9evXt73mzb62drx58+Y88MAD2W233d7rWwSgQWytBIAucuyxx2b27Nl57rnnkiQvvPBCnnrqqe2+rm/fvunfv38WLFiQJPnlL3/Ztjq366675vjjj8+XvvSlfOELX0iSttC211575ZVXXsns2bPfdr1Zs2YlSe6777707ds3ffv2fdvz48aNy49+9KO24ze3bgKw47IiBwBdZODAgbn00kszbty4bN68Oe973/vy4x//eJuveXO17Lrrrsu5556btWvX5sADD8z06dPb2nzuc5/LnDlzMm7cuCRJv379cs4552TIkCH58Ic/3LaV8039+/fP4YcfnjVr1uTnP//5u/r84Q9/mPPPPz9Dhw7Nxo0b09zcnKuvvrqjbx+ALlTeelerHU1TU1O1pe++AYCe6Mtf/nJGjBjRttK2NdOmTctLL72Ub3/7291UGQCNUkpZUlVV0zvPW5EDgB3AN7/5zfzhD3/IxRdfvM1248ePz8qVK3P33Xd3T2EA7JCsyAEAAOygtrYi52YnAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADXToSBXSplQSnmklLK5lPKuO6m8pd0JpZTHSylPlFK+0ZE+AQAAeruOrsg9nOTTSe7dWoNSSp8kP05yYpKBSSaWUgZ2sF8AAIBeq0NfCF5V1WNJUkrZVrPDkjxRVdWTrW1/k+RTSR7tSN8AAAC9VXd8Ru7fk/ztLcdPt57bolLKlFLK4lLK4tWrV3d5cQAAAHWz3RW5Usr8JPtt4amLqqq6qR19bGm5rtpa46qqrklyTZI0NTVttR0AAEBvtd0gV1XVJzrYx9NJPvSW4/9I8kwHrwkAANBrdcfWykVJDiqlDCil7JzkjCS/64Z+AQAAeqSOfv3A+FLK00nGJLm1lHJ76/n9SynzkqSqqo1Jpia5PcljSf6zqqpHOlY2AABA79XRu1bOTTJ3C+efSXLSW47nJZnXkb4AAAB4Q3dsrQQAAKATCXIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjHVRPs0AAAmpSURBVCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAOxA9thjjy69/qRJkzJ79uwu7QOArifIAQAA1IwgBwA7oKqqcuGFF2bw4MEZMmRIZs2alSQ5/fTTM2/evLZ2kyZNyo033phNmzblwgsvzKhRozJ06ND89Kc/bbvO1KlTM3DgwJx88sl57rnnGvJ+AOhcOzW6AADg3ebMmZOWlpYsXbo0zz//fEaNGpXm5uacccYZmTVrVk466aS89tprueuuu3LVVVfl2muvTd++fbNo0aJs2LAhY8eOzbhx4/LnP/85jz/+eP7yl7/k2WefzcCBA3P22Wc3+u0B0EGCHADsgO67775MnDgxffr0yb777psjjzwyixYtyoknnpivfOUr2bBhQ2677bY0Nzdnt912yx133JGHHnqo7fNvL730UlasWJF777237Tr7779/jjnmmAa/MwA6gyAHADugqqq2eH7XXXfNUUcdldtvvz2zZs3KxIkT29pfeeWVOf7449/Wft68eSmldHm9AHQvn5EDgB1Qc3NzZs2alU2bNmX16tW59957c9hhhyVJzjjjjEyfPj0LFixoC27HH398rrrqqrz++utJkuXLl+fVV19Nc3NzfvOb32TTpk35+9//nnvuuadh7wmAzmNFDgB2QOPHj88DDzyQYcOGpZSSyy+/PPvtt1+SZNy4cTnzzDNz6qmnZuedd06STJ48OatWrcqIESNSVVX23nvv/Pa3v8348eNz9913Z8iQITn44INz5JFHNvJtAdBJyta2buwImpqaqsWLFze6DAAAgIYopSypqqrpnedtrQQAAKgZQQ4AAKBmBDkAAICaEeQAAABqRpADAACoGUEOAACgZgQ5AACAmhHkAAAAakaQAwAAqBlBDgAAoGYEOQAAgJoR5AAAAGpGkAMAAKgZQQ4AAKBmBDkAAICaEeQAAABqRpADAACoGUEOAACgZgQ5AACAmhHkAAAAakaQAwAAqBlBDgAAoGYEOQAAgJoR5AAAAGpGkAMAAKgZQQ4AAKBmBDkAAICaEeQAAABqRpADAAComQ4FuVLKhFLKI6WUzaWUpm20W1VK+UsppaWUsrgjfQIAAPR2O3Xw9Q8n+XSSn7aj7dFVVT3fwf4AAAB6vQ4FuaqqHkuSUkrnVAMAAMB2dddn5Kokd5RSlpRSpmyrYSllSillcSll8erVq7upPAAAgPrY7opcKWV+kv228NRFVVXd1M5+xlZV9UwpZZ8kd5ZSllVVde+WGlZVdU2Sa5Kkqampauf1AQAAeo3tBrmqqj7R0U6qqnqm9edzpZS5SQ5LssUgBwAAwLZ1+dbKUsrupZQ933ycZFzeuEkKAAAA/4KOfv3A+FLK00nGJLm1lHJ76/n9SynzWpvtm+S+UsrSJH9McmtVVbd1pF8AAIDerKN3rZybZO4Wzj+T5KTWx08mGdaRfgAAAPgv3XXXSgAAADqJIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNCHIAAAA1I8gBAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM0IcgAAADUjyAEAANSMIAcAAFAzghwAAEDNlKqqGl3DVpVSVid5qtF1JNkryfONLoJuZ9x7J+PeOxn33sm4907GvXeq87gfUFXV3u88uUMHuR1FKWVxVVVNja6D7mXceyfj3jsZ997JuPdOxr136onjbmslAABAzQhyAAAANSPItc81jS6AhjDuvZNx752Me+9k3Hsn49479bhx9xk5AACAmrEiBwAAUDOCHAAAQM0Icu9QSplQSnmklLK5lLLVW5SWUlaVUv5SSmkppSzuzhrpfO9h3E8opTxeSnmilPKN7qyRrlFK+W+llDtLKStaf/bfSjtzvua2N3/LG37Y+vxDpZQRjaiTztWOcT+qlPJS69xuKaX8z0bUSecqpfy8lPJcKeXhrTxvvvdA7Rj3HjXfBbl3ezjJp5Pc2462R1dVNbynfSdFL7XdcS+l9Eny4yQnJhmYZGIpZWD3lEcX+kaSu6qqOijJXa3HW2PO11Q75++JSQ5q/W9Kkqu6tUg63Xv4d3tB69weXlXVt7q1SLrKjCQnbON5871nmpFtj3vSg+a7IPcOVVU9VlXV442ug+7VznE/LMkTVVU9WVXVa0l+k+RTXV8dXexTSa5rfXxdkv/ewFroOu2Zv59K8ovqDQ8m6VdK+bfuLpRO5d/tXqqqqnuTvLCNJuZ7D9SOce9RBLl/XZXkjlLKklLKlEYXQ7f49yR/e8vx063nqLd9q6r6e5K0/txnK+3M+Xprz/w1x3ue9o7pmFLK0lLK/ymlDOqe0mgw87336jHzfadGF9AIpZT5SfbbwlMXVVV1UzsvM7aqqmdKKfskubOUsqz1/wKwg+qEcS9bOOf7O2pgW2P/Hi5jztdbe+avOd7ztGdM/5TkgKqqXimlnJTkt3ljux09m/neO/Wo+d4rg1xVVZ/ohGs80/rzuVLK3LyxfcMvdTuwThj3p5N86C3H/5HkmQ5ek26wrbEvpTxbSvm3qqr+3rqt5rmtXMOcr7f2zF9zvOfZ7phWVbXmLY/nlVJ+UkrZq6qq57upRhrDfO+Fetp8t7XyX1BK2b2Usuebj5OMyxs3y6BnW5TkoFLKgFLKzknOSPK7BtdEx/0uyVmtj89K8q7VWXO+R2jP/P1dkjNb72Y3OslLb267pba2O+6llP1KKaX18WF543ej/9ftldLdzPdeqKfN9165IrctpZTxSa5MsneSW0spLVVVHV9K2T/J/66q6qQk+yaZ2/r3YKckv66q6raGFU2HtWfcq6raWEqZmuT2JH2S/LyqqkcaWDad4/tJ/rOU8j+S/N8kE5LEnO9ZtjZ/Synntj5/dZJ5SU5K8kSStUm+0Kh66RztHPfPJPlSKWVjknVJzqiqyha7miulXJ/kqCR7lVKeTvK/krwvMd97snaMe4+a76XGtQMAAPRKtlYCAADUjCAHAABQM4IcAABAzQhyAAAANSPIAQAA1IwgBwAAUDOCHAAAQM38f7BvLg32VYm7AAAAAElFTkSuQmCC\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "embedding_matrix = classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights')\n",
    "\n",
    "for term_index in range(len(informative_terms)):\n",
    "  # Create a one-hot encoding for our term.  It has 0s everywhere, except for\n",
    "  # a single 1 in the coordinate that corresponds to that term.\n",
    " # 为我们的术语创建一个一热编码(独热编码)。它到处都是0，除了在对应这一项的坐标中只有一个1。\n",
    "  term_vector = np.zeros(len(informative_terms))\n",
    "  term_vector[term_index] = 1\n",
    "  # We'll now project that one-hot vector into the embedding space.\n",
    "    # 现在我们把这个热点向量投影到嵌入空间。\n",
    "  embedding_xy = np.matmul(term_vector, embedding_matrix)\n",
    "  plt.text(embedding_xy[0],\n",
    "           embedding_xy[1],\n",
    "           informative_terms[term_index])\n",
    "\n",
    "# Do a little setup to make sure the plot displays nicely.\n",
    "# 做一些设置以确保情节显示得很好。设置图表参数\n",
    "plt.rcParams[\"figure.figsize\"] = (15, 15)\n",
    "plt.xlim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
    "plt.ylim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
    "plt.show() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pUb3L7pqLS86",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 任务 6：尝试改进模型的效果\n",
    "\n",
    "看看您能否优化该模型以改进其效果。您可以尝试以下几种做法：\n",
    "\n",
    "* **更改超参数**或**使用其他优化工具**，比如 Adam（通过遵循这些策略，您的准确率可能只会提高一两个百分点）。\n",
    "* **向 `informative_terms` 中添加其他术语。**此数据集有一个完整的词汇表文件，其中包含 30716 个术语，您可以在以下位置找到该文件：https://download.mlcc.google.cn/mledu-datasets/sparse-data-embedding/terms.txt 您可以从该词汇表文件中挑选出其他术语，也可以通过 `categorical_column_with_vocabulary_file` 特征列使用整个词汇表文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "6-b3BqXvLS86",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://download.mlcc.google.cn/mledu-datasets/sparse-data-embedding/terms.txt\n",
      "253952/253538 [==============================] - 0s 0us/step\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
     ]
    }
   ],
   "source": [
    "# Download the vocabulary file.\n",
    "terms_url = 'https://download.mlcc.google.cn/mledu-datasets/sparse-data-embedding/terms.txt'\n",
    "terms_path = tf.keras.utils.get_file(terms_url.split('/')[-1], terms_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "0jbJlwW5LS8-",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D6A17978>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D6A17978>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D689E978>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D689E978>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D689EE80>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D689EE80>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D689EDD8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D689EDD8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D69CE208>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D69CE208>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D61AF2E8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D61AF2E8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D71C1E10>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D71C1E10>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D71C19E8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D71C19E8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D71C1908>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D71C1908>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D71C1518>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D71C1518>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Training set metrics:\n",
      "accuracy 0.89744\n",
      "accuracy_baseline 0.5\n",
      "auc 0.9579011\n",
      "auc_precision_recall 0.95729756\n",
      "average_loss 0.26320788\n",
      "label/mean 0.5\n",
      "loss 6.580197\n",
      "precision 0.8939106\n",
      "prediction/mean 0.50254256\n",
      "recall 0.90192\n",
      "global_step 10000\n",
      "---\n",
      "WARNING: Entity <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D39CE780>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method _DNNModel.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModel object at 0x000001E1D39CE780>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D6354518>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method DenseFeatures.call of <tensorflow.python.feature_column.feature_column_v2.DenseFeatures object at 0x000001E1D6354518>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D6354A20>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D6354A20>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D6354978>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D6354978>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D6354D68>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E1D6354D68>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "Test set metrics:\n",
      "accuracy 0.8672\n",
      "accuracy_baseline 0.5\n",
      "auc 0.93779194\n",
      "auc_precision_recall 0.9350573\n",
      "average_loss 0.3201093\n",
      "label/mean 0.5\n",
      "loss 8.002733\n",
      "precision 0.8673763\n",
      "prediction/mean 0.49760142\n",
      "recall 0.86696\n",
      "global_step 10000\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "# Create a feature column from \"terms\", using a full vocabulary file.\n",
    "informative_terms = None\n",
    "with io.open(terms_path, 'r', encoding='utf8') as f:\n",
    "  # Convert it to a set first to remove duplicates.\n",
    "  informative_terms = list(set(f.read().split()))\n",
    "  \n",
    "terms_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key=\"terms\", \n",
    "                                                                                 vocabulary_list=informative_terms)\n",
    "\n",
    "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=20)\n",
    "feature_columns = [ terms_embedding_column ]\n",
    "\n",
    "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.01)\n",
    "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
    "\n",
    "classifier = tf.estimator.DNNClassifier(\n",
    "  feature_columns=feature_columns,\n",
    "  hidden_units=[10,10],\n",
    "  optimizer=my_optimizer\n",
    ")\n",
    "\n",
    "classifier.train(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=10000)\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([train_path]),\n",
    "  steps=1000)\n",
    "print(\"Training set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")\n",
    "\n",
    "evaluation_metrics = classifier.evaluate(\n",
    "  input_fn=lambda: _input_fn([test_path]),\n",
    "  steps=1000)\n",
    "\n",
    "print(\"Test set metrics:\")\n",
    "for m in evaluation_metrics:\n",
    "  print(m, evaluation_metrics[m])\n",
    "print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ew3kwGM-LS9B",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## 总结\n",
    "\n",
    "我们可能获得了比我们原来的线性模型更好且具有嵌入的 DNN 解决方案，但线性模型也相当不错，而且训练速度快得多。线性模型的训练速度之所以更快，是因为它们没有太多要更新的参数或要反向传播的层。\n",
    "\n",
    "在有些应用中，线性模型的速度可能非常关键，或者从质量的角度来看，线性模型可能完全够用。在其他领域，DNN 提供的额外模型复杂性和能力可能更重要。在定义模型架构时，请记得要充分探讨您的问题，以便知道自己所处的情形。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9MquXy9zLS9B",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ### *可选内容：*在 `embedding_column` 与 `indicator_column` 之间进行权衡\n",
    "\n",
    "从概念上讲，在训练 `LinearClassifier` 或 `DNNClassifier` 时，需要根据实际情况使用稀疏列。TF 提供了两个选项：`embedding_column` 或 `indicator_column`。\n",
    "\n",
    "在训练 LinearClassifier（如**任务 1** 中所示）时，系统在后台使用了 `embedding_column`。正如**任务 2** 中所示，在训练 `DNNClassifier` 时，您必须明确选择 `embedding_column` 或 `indicator_column`。本部分通过一个简单的示例讨论了这两者之间的区别，以及如何在二者之间进行权衡。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M_3XuZ_LLS9C",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 假设我们的稀疏数据包含 `\"great\"`、`\"beautiful\"` 和 `\"excellent\"` 这几个值。由于我们在此处使用的词汇表大小为 $V = 50$，因此第一层中的每个单元（神经元）的权重将为 50。我们用 $s$ 表示稀疏输入中的项数。对于此示例稀疏数据，$s = 3$。对于具有 $V$ 个可能值的输入层，带有 $d$ 个单元的隐藏层需要运行一次“矢量 - 矩阵”乘法运算：$(1 \\times V) * (V \\times d)$。此运算会产生 $O(V * d)$ 的计算成本。请注意，此成本与隐藏层中的权重数成正比，而与 $s$ 无关。\n",
    "\n",
    "如果输入使用 [`indicator_column`](https://www.tensorflow.org/api_docs/python/tf/feature_column/indicator_column) 进行了独热编码（长度为 $V$ 的布尔型矢量，存在用 1 表示，其余则为 0），这表示很多零进行了相乘和相加运算。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I7mR4Wa2LS9C",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 当我们通过使用大小为 $d$ 的 [`embedding_column`](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column) 获得完全相同的结果时，我们将仅查询与示例输入中存在的 3 个特征 `\"great\"`、`\"beautiful\"` 和 `\"excellent\"` 相对应的嵌入并将这三个嵌入相加：$(1 \\times d) + (1 \\times d) + (1 \\times d)$。由于不存在的特征的权重在“矢量-矩阵”乘法中与 0 相乘，因此对结果没有任何影响；而存在的特征的权重在“矢量-矩阵”乘法中与 1 相乘。因此，将通过嵌入查询获得的权重相加会获得与“矢量-矩阵”乘法相同的结果。\n",
    "\n",
    "当使用嵌入时，计算嵌入查询是一个 $O(s * d)$ 计算；从计算方面而言，它比稀疏数据中的 `indicator_column` 的 $O(V * d)$ 更具成本效益，因为 $s$ 远远小于 $V$。（请注意，这些嵌入是临时学习的结果。在任何指定的训练迭代中，都是当前查询的权重。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "etZ9qf0kLS9D",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " 正如我们在**任务 3** 中看到的，通过在训练 `DNNClassifier` 过程中使用 `embedding_column`，我们的模型学习了特征的低维度表示法，其中点积定义了一个针对目标任务的相似性指标。在本例中，影评中使用的相似术语（例如 `\"great\"` 和 `\"excellent\"`）在嵌入空间中彼此之间距离较近（即具有较大的点积），而相异的术语（例如 `\"great\"` 和 `\"bad\"`）在嵌入空间中彼此之间距离较远（即具有较小的点积）。\n",
    "read end by hank"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "mNCLhxsXyOIS",
    "eQS5KQzBybTY",
    "copyright-notice"
   ],
   "default_view": {},
   "name": "intro_to_sparse_data_and_embeddings.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}